Dual Graph Complementary Network
Authors: Chenhua Liu
Kun Zhan
Venue:  (2021)

Abstract: As a powerful representation learning method on graph data, graph neural networks (GNNs) have shown great popularity in tackling graph analytic problems. Although many attempts have been made in literatures to find strategies about extracting better embedding of the target nodes, few of them consider this issue from a comprehensive perspective. Most of current GNNs usually employ some single method which can commendably extract a certain kind of feature but some equally important features are often ignored. In this paper, we develop a novel dual graph complementary network (DGCN) to learn representation complementarily. We use two different branches, and inputs of the two branches are the same, which are composed of structure and feature information. At the same time, there is also a complementary relationship between the two branches. Beyond that, our extensive experiments show that DGCN outperforms state-of-the-art methods on five public benchmark datasets.

Under review as a conference paper at ICLR 2021
DUAL GRAPH COMPLEMENTARY NETWORK
Anonymous authors
Paper under double-blind review
ABSTRACT
As powerful representation learning method on graph data, graph neural net- works (GNNs) have shown great popularity in tackling graph analytic problems. Although many attempts have been made in literatures to find strategies about extracting better embedding of the target nodes, few of them consider this issue from a comprehensive perspective. Most of current GNNs usually employ some single method which can commendably extract certain kind of feature but some equally important features are often ignored. In this paper, we develop a novel dual graph complementary network (DGCN) to learn representation complemen- tarily. We use two different branches, and inputs of the two branches are the same, which are composed of structure and feature information. At the same time, there is also a complementary relationship between the two branches. Beyond that, our extensive experiments show that DGCN outperforms state-of of-the-art methods on five public benchmark datasets.
INTRODUCTION
Although many attempts have been made in literatures find a better strategy learn the target node representation, the feature extraction capabilities of most methods are still far from optimal, especially when only a small amount of data is labeled. However, fact, compared with the expen- sive and laborious acquisition of labeled data, unlabeled data is much easier to obtain. Therefore, how to learn more useful representations with limited label information is the key direct of repre- sentation learning study. Methods of this issue, commonly referred to as semi-supervised learning, which essentially believe that the similar points have similar outputs. Thus, it can properly utilize the consistency of data to make full use of the rich information of unsupervised data.
In the real world, it is common that we have data with specific topological structures which usually called graph data. The graph structure is usually expressed the connection between nodes. By aggregating the features of neighborhood and performing appropriate linear transformation, graph neural networks (GNNs) can convert graph data into Low-dimensional, compact, and continuous feature space. Nevertheless, most of them only care about a single aggregation strategy, which is counter intuitive: for example, far as social networks are concerned, the relationship between people is very complex, while, most of the traditional GNNs only consider the single connection between nodes and ignore other implicit information.
In this paper, our work focuses on learning node representations by GNNs in semi-supervised way. Despite there are already many graph-based semi-supervised learning methods (Kıpí & Welling 2016: Yangg et all, 2016 Khan & Blumenstock, 2019), most of them can only find a single rela- tionship between nodes. As a result, some information in unsupervised data is usually ignored. To overcome this problem, we develop a novel dual graph complementary network (DGCN) to extract information from both feature and topology spaces. An intuition of our method is to learn based on disagreement: network performance is largely related to the quality of the graph, which usually emphasizes the relevance of an attribute of instances. So, since we don't t know what attributes are most important, consider both of them in the model design.
Compared with the traditional GNN-based methods, we perform two different aggregate strategies which emphasize different attributes in each branch, from the perspective of node feature, and the other from the topological structure. Then, to further utilize implicit information, we employ two networks with different structures extract embedding from input feature. By doing so, nodes'

---

Under review as a conference paper at ICLR 2021
information can be propagated in different ways. Then, the supervised loss lxup and diversity straint Ediv are used to guide the training.
We use two different branches to extract common information in topology and feature spaces. By utilizing disagreements between the two branches, model can gain information that may be ignored by single branch.
To prove the effectiveness of our method, we conducted experiments on five public benchmark datasets.
The contributions of our work are summarized as follows:
We propose a novel dual graph complementary network (DGCN) to fuse complementary information, which utilizes different graphs to aggregate nodes that are similar in certain attributes in a complementary way.
By comparing with algorithms that use non-single graphs, it proves that our complementary architecture can extract richer information
Through extensive evaluation multiple datasets, demonstrate DGCN effectiveness over state-of-the--ee baselines.
2
RELATED WORK
2.1 SEMI-SUPERVISED LEARNING
Semi-supervised learning is usually aimed at the case of insufficient data labels. X E Raxd the feature of input nodes. Y [yy] E Raxk is the label matrix, where & is the class number. Vij means that the i-th node belongs to the j-th class. Then split data points into labeled and unlaheled points. Accordingly, æ and æy express feature of labeled and unlabeled instance, respectively. Moreover, the ground-trui truth label of the label nodes is available only.
The main objective of semi-supervised learning is extract supervised information from labeled dataset whilst adequately utilizing data distribution information contained in X. There are four categories of sumi-supervised learning algorithms:
Self-training semi cemi-supervised learning LLec 2013): It utilizes high-confidence pseudo la- bels to expand label set. Ideally, it can continuously improve network performance, but is usually limited by the quality of pseudo labels.
2. Graph-based semi-supervised learning: It propagates information between instances ac- cording to edges in graph. It's inductive learning method, of which the performance mainly depends on the aggregation algorithm.
3. Low-density separation methods (Joachims 1999): They assume that the decision hyper- plane is consistent with the data distribution, of the data.
should pass through the sparse region
4. Pretrain semi-supervised learning: such as autoencoder (Vincent et all 2008; Ritai et all 2011), trains the model based reconstruction error and then fine tune it using labeled data.
However, semi-supervised learning tasks prefer obtain information related to data distribution rather than all information of samples. this paper, we mainly focus on the graph-based semi- supervised learning.
2.2 GRAPH- BASED REMI-SUPERVISER LEARNING
In addition to features, graph- semi-supervised learning methods (Kıpī & Welling 2016) rep- resent the topological edge connection between different instances. For many datasets, graph is given feature. If the features of the dataset do not contain the relationships between different samples, graph can also be constructed by measuring the similarity between the features of the instances (Zhu et all 2003). Actually, the graph is measure of whether the instances closely
2

---

Under review as a conference paper at ICLR 2021
connected. Then, according to this graph, information exchange between instances be carried out, so that the information largely related to the quality of the graph. When the attributes emphasized in the graph do not match the expectations of the task objective, misjudgments are often caused. Usually, is difficult to find- ing what really matters. The traditional graph-based semi-supervised learning methods usually uses a single graph for node aggregation, which causes a single attribute to be emphatically considered, but when this attribute does not match the task goal, it will mislead the training instead.
unlabeled data can be effectively utilized. Network performance is
3 DGCN ARCHITECTURE
In this section, will present the overall framework of DGCN, see Fig. M. The main idea of DGCN that information exchange under the control of graphs emphasizing different attributes can extract more abundant features. To this end, use branches extract information from two inputs the same time. The node features these two inputs are the same, the only difference is the graphs that control the information exchange. In addition, in order to further expand the difference between branches, we use a diversity loss laiv.
H
Attention
Hgen
Viewer
Hgen 2.1
Attention
Ediv
H
MLP
Output
Attention
Heat
P
Viewer
A2
Figure I:
The framework of DGCN network. The original dataset contains the graph and the feature. First, the node features in the dataset construct another graph, then use viewer and viewer 2 observe the graphs at the same time, and the latent features are H Hgen H and F respectively. Then, we fuse GCN view and GAT view respectively to obtain Hgen and H respectively through attention operation. The obtained Hgen and Hgat are sent to the final attention layer together with the previous Hgen Hgan H and Hgat
3.1 NOTATION & PROBLEM STATEMENT
Let & = (V, A, X') be an undirected graph. V is the set of nodes on the graph, which is composed of unlabeled (Va)) and labeled (Vi) nodes with the number of nodes is n, and nı respectively. ny + nu is the number of nodes. A = [a:z] E Raxn the adjacency matrix. dij = 1 represents that node and node are closely related in attribute, otherwise, aij-
3.2 BRANCHES
In order to capture different characteristics by the two branches (also called viewer), we use different network structures for each branch: GCN (Kipi & Welling, 2016 and GAT (Vehčković et all 20L7). Given a graph G = (V,A, A'), both GCN and GAT intend to extract richer features at vertex by aggregating features of vertices from its neighborhood (Li 2019). So the node representation
3

---

Under review as a conference paper at ICLR 2021
of the 1-th layer Hj can be defined by:
H1 Updaatt(AsgrgaattoHH.....
Oupdate).
where 0 and @lpdaae are the learnable weights of aggregation and update functions of the 1-th layer respectively and the initial Hg = X. The aggregation and update functions are the essential components of GNNs, and obviously the features extracted by different aggregation functions will have certain differences. Thus, we take advantage of two different networks, GCN and GAT, to obtain node representation.
The node features output by the I-th GCN layer can be expressed as:
H
1) -1)
where I E 1 indicates the identity matrix. A + 1 means adding self-loop in the graph, D is the diagonal degree matrix of A + I, and (-)) the activation functiin.
It can be seen from equation @ that GCN aggregates neighbor features by weighting the value of symmetric normalized laplacian.
Next, we introduce the algorithm GAT that uses the attention mechanism to calculate the neighbor weight. Through learnable coefficient a. GAT can assign learnable weights to each neighbor of the node. For node i, the weight Oij between it and its neighbor node j can be expressed as:
exp (LeakyRoLU [Wh,
au EREN; exp LeakyReLUU ( [W h, [ Whhl))
where is the transposition operation and || represents concatenation. Then the forward propaga- tion process of node v in I-th layer can be represented as:
11 = 11 M
1 o
of W W.Thhii JEN:
Σ
where, h4s is the embeding of node i in the 1-th layer. M is the number of independent attention mechanisms. a is activation function of GAT. ox is the normalized attention coefficients computed by the m-th attention mechanism. see equation BL As can be seen from equation HL the weights GAT assigns to a node's neighbors are learnable. Thus we can assign adaptive weights to different neighbors.
Although these two methods are based on the existence of connection between points as the premise of aggregation. Both the GCN and GAT models tages. The former considers the relationship between nodes (probability conduction matrix), but can't learn neighbor weights dynamically. Although the latter assign dynamic weights to neigh- bors, ignores the influence of degree attribute of node on aggregation. Therefore, using these two branches, extract more complementary features from the input.
have their advantages and disadvan-
3.3 FORWARD PROPAGATION
In this subsection, we introduce the input used by the network and the specific forward propagation strategy. In order consider different attributes when aggregating, we use different graphs for training. but adopt the same features. In this paper, the datasets used in experiment are graph- structured which have two characteristics, one the feature of the instance itself, which is not affected by other instances, and the other reflects the relationship with other instances.
For example, dataset ACM (Wang et all 2019) which extracted from ACM dataset contains 3025 papers. It has two properties: one is a bag-of-worte that indicates whether the keyword exists, and the other indicates which papers are written by the same author. obviouul we only base whether the paper is written by the same author as the basis for aggregation, will inevitably
4

---

Under review as a conference paper at ICLR 2021
ignore the situation where the same author has written different types of papers and the same type of papers belong to different authors, thus mistakenly aggregate articles of different categories together. Therefore, also construct graph based another attribute of the dataset: bag-of-worde information can be transferred between instances with similar keywords. In order to measure the similarity of instances' features, find the cosine similarity between the features of all instances:
that
3. Fjj z Øz
where Sij denotes the cosine similarity between the feature æ; of node i and the feature æ, of node j E
Notice that j o i. For node i, we choose the & largest 8ij and let the corresponding as the neighbors of i. Then if j is the neighbor of node i, obviously is the neighbor of node too. As above-mentions mentioned. represent the inherent graph structure of the data and the graph constructed according to the feature, respectively.
get
graph constructed from features. We use A1 and Ag to
Therefor, by inputting A₁ and A2 for each branch, we can get four different outputs. according to equation [ and equation g the forward propagation of DGCN can be represented as:
1 = o ((n (A. + 1)D )
uer- 0n1)
(6)
gat o.lei
I a
Σ JEN;
Wi gus
(7)
where, v represents that the graph A1, while = 2 corresponding to Ag. a and are the activation function. a.,,, is the normalized attention coefficients. O.,1 and W the weights of linear transformations. For the GAT branch, h n,I.; means the representation of node in the I-th layer with the input graph is A,- Similarly, Heth corresponds to the l-layer embedding matrix of the GCN branch when the input graph is A..
For these four embeddings, first use the attention mechanism combine the embeddings gener- ated by different graphs of the same branch:
H = att H HgT
(8)
H att H |
HH2
(9)
Then, apply the attention mechanism again combine H1 11 11 H1 11 and H Through these two attention operations, dynamically assign weights different embedding to find attributes that better match the task goal.
3.4
Loss FUNCTIONS OF DGCN
The objective function DGCN consists of two parts: the supervised loss Sup and the diversity loss laiv-
3.4.1
SUPERVISED LOSS
Given graph G = (,, X), as V = VVV,, the corresponding label of Vi is Y. In order to utilize the supervision information, we use the cross-entropy loss function to guide the training:
loup = Σ
Σ
Vig In Paj-
(10)
SEV; j=1
5

---

Under review as a conference paper at ICLR 2021
where Vi is the ground-trute truth label of node and Pij is the model predicted label. k is the number of classes.
3.4.2 DIVERSITY LOSS
In order further expand the differences between branches and capture richer node features, use Laiee to add a diversity constraint on Hg and Hg First, we use I=g-normalization to nor- malize Ha and H output by the attention module. The normalized results are Aron and Heat respectively. Using the above results, we can capture the similarity node embedding:
$ Argnn (frgat)
(11)
Then, the diversity loss can be defined by:
Σ Σ3 2j=1 $6
laiv
(12)
where is the number of nodes. Through this diversity constraint, we can expand the difference between the branches to learn complementary features. Therefore, our final optimization object can be expressed as:
hotal = (1 - s)p + ylaiv
(13)
where y is parameter of the disparity constraint terms. Using this objective function, we can optimize the proposed model through back propagation and learn the node embedding for classification.
Table Statistics of the datasets. Refer Section for more details.
Datasets
Nodes
Edges
Classes
Features
Training
Vi
Test
ACM
3025
13128
3
1870
60/120/180
0.020/0.040/0.060
1000
UA12010
3067
28311
19
4973
380/760/1 140
0.124/0.248/0.377
1000
Citeseer
3327
4732
3703
120/240/360
(003//0002/0..0 108
1000
BlogCatalog
5196
171743
6
8189
120/240/360
0.023/0.
3.023/0.046/0..00 046/0.069
1000
Flickr
7575
239738
12047
180/360/540
0.024/0.048/0.000
1000
EXPERIMENT
4.1 DATASETS
For evaluating the effectiveness of DGCN, we evaluate several semi-supervised classification benchmarks. Following the experimental setup of [aan et al. (2020), we evaluate on five datasets.
ACM (Wang et all 2019): This dataset is extracted from the ACM dataset, where the nodes represent the papers, the edges represent that the connected two papers belong to the same author, and the feature is the word bag representation of paper's keywords.
UA12010 (Wang all 2018): This dataset has 3067 nodes and classes.
Citeseer (Kıpf & Welling 201G): In the CiteSeer dataset, papers are divided into six cate-
gories, containing total 3312 papers, which record citation information between papers. And the feature is the word bag representation of the paper.
BlogCatalog (Meng et all, 2019: This is a network of social relationships from the Blog- Catalog websitewhere the nodes are bloggers and edges their social relationships. Node attributes are the short descriptions of users' blogs provided by users. The labels represent the topic categories provided by the authors which can divided into 6 classes.
Flickr (Meng et all, 2019): This network is built from profile and relation data of users in Flickr. We treat each user as node, relationships between two user as an edge. The labels represent the interest groups of the users.
The detailed descriptions of the datasets used here shown in Table III

---

Under review as a conference paper at ICLR 2021
Table 2: Experiments results (%) on the node classification task. L/C means the number of labeled nodes per class
L/C
20
40
60
Datasets
Metries
ACC
F1
ACC
F1
ACC
F1
Deep Walk
62.69
62.11
63.00
61.88
67.03
66.99
LINE
41.28
40.12
45.83
45.79
50.41
49.92
GCN
87.80
87.82
89.06
89.00
90.54
90.49
kNN-GCN
78.52
78.14
81.66
81.53
82.00
81.95
ACM
GAT
87.36
87.44
88.60
88.55
90.40
90.39
DEMO-Net
84.48
84.16
85.70
84.83
86.55
84.05
MixHop
81.08
81.40
82.34
81.13
83.09
82.24
AM-GCN
90.40
90.43
90.76
90.66
91.42
91.36
DGCN
91.10
91.07
91.40
91.35
91.90
91.90
Deep Walk
42.02
32.93
51.26
46.01
54.37
44.43
LINE
43.47
37.01
45.37
39.62
51.05
43.76
GCN
49.88
32.86
51.80
33.80
54.40
34.12
kNN-GCN
66.06
52.43
68.74
54.45
71.64
54.78
UA12010
GAT
56.92
39.61
63.74
45.08
68.44
48.97
DEMO-Net
23.45
16.82
30.29
26.36
34.11
29.05
MixHop
61.56
49.19
65.05
53.86
67.66
56.31
AM-GCN
70.10
55.61
73.14
64.88
74.40
65.99
DGCN
72.50
58.57
75.80
65.89
78.00
70.19
DeepWalk
43.47
38.09
45.15
43.18
48.86
48.01
LINE
32.71
31.75
33.32
32.42
35.39
34.37
GCN
70.30
67.50
73.10
69.70
74.48
71.24
kNN-GCN
61.35
58.86
61.54
59.33
62.38
60.07
Citeseer
GAT
72.50
68.14
73.04
69.58
74.76
71.60
DEMO-Net
69.50
67.84
70.44
66.97
71.86
68.22
MixHop
71.40
66.96
71.48
67.40
72.16
69.31
AM-GCN
73.10
68.42
74.70
69.81
75.56
70.92
DGCN
74.60
69.46
75.30
71.14
76.90
72.97
Deep Walk
38.67
34.96
50.80
48.61
55.02
53.56
LINE
58.75
57.75
61.12
60.72
64.53
63.81
GCN
69.84
68.73
71.28
70.71
72.66
71.80
kNN-GCN
75.49
72.53
80.84
80.16
82.46
81.90
BlogCatalog
GAT
64.08
63.38
67.40
66.39
69.95
69.08
DEMO-Net
54.19
52.79
63.47
63.09
76.81
76.73
MixHop
65.46
64.89
71.66
70.84
77.44
76.38
AM-GCN
81.98
81.36
84.94
84.32
87.30
86.94
DGCN
88.70
88.31
90.30
90.02
92.00
91.69
DeepWalk
24.33
21.33
28.79
26.90
30.10
27.28
LINE
33.25
31.19
37.67
37.12
38.54
37.77
GCN
41.42
39.95
45.48
43.27
47.96
46.58
KNN-GCN
69.28
70.33
75.08
75.40
77.94
77.97
Flickr
GAT
38.52
37.00
38.44
36.94
38.96
37.35
DEMO-Net
34.89
33.53
46.57
45.23
57.30
56.49
MixHop
39.56
40.13
55.19
56.25
64.96
65.73
AM-GCN
75.26
74.63
80.06
79.36
82.10
81.81
DGCN
74.6
72.47
81.1
83.4
81.06
83.18

---

Under review as a conference paper at ICLR 2021
4.2 BASELINES
We compare with some state-of-art baselines to verfify the effectiveness of the proposed DGCN.
DeepWalk (Perozzi 2014) a random walk based network embedding method, learn- ing feature by treating truncated random walks in a graph as the equivalent of sentences.
LINE (Tang all (015)) is large-scale embedding method that retains both the local network structure and the global network structure.
GCN (Kipf & Welling, 20TG) is a variant of convolutional neural networks which aggre- gates information of nodes get node characteristics.
KNN-GCN, The network structure of KNN-GCN is the same as that of GCN. But the graph we use here is the aforementioned A2, see Section B3.
GAT (Velíčkovíć et all 2017) is graph attention based method which can assign different weights to nodes during aggregation.
DEMO-Net (Wu 2019) assumes that nodes with the same degree value will share the same graph convolution, and the feature aggregation is expressed as multi-task learning problem according to the degree value the node.
MixHop (Abu-EI-Harja et al, 2019) can learn the neighbor mixture relationship by repeat- edly mixing the feature representations of neighbors at various distances.
AM-GCN (Wang all 2020) extracts embeddings from node features, topological struc- tures and their combinations, and uses the attention mechanism to learn the adaptive im- portance weights of embeddings
4.3 RESULTS
We train the DGCN network described in Section Bon five public datasets and evaluate the prediction accuracy on a test set of 1, 1,000) labeled examples, and experiments on all datasets are optimized using Adam optimizer. The model adopts GCN and GAT branches with layer number of 2, and the quantitative analysis results can be seen in Table
It can seen from the Table that DGCN can exceed the baseline in most of the accuracy rates of all datasets, which proves the effectiveness of method. On most datasets, the performance of DGCN is better than AM-GCN using two graphs and GCN, kNN- GCN, GAT using one graph, which fully proves that DGCN capture more information that meets the task objectives.In addition, by comparing with AM-GCN, which also uses different graphs for learning, our DGCN can leam better node embeddings through its complementary learning mechanism.
The main difference between GCN and KNN-GCN is that the structure graph and cosine graph are used respectively. For dataset, graph that is more relevant its classification goal(Nodes connected by edges are more likely to belong to the same class) will perform better. As we can see, on UAI2010, BlogCatelog and Flickr kNN-GCN seem to be signif- icantly better than GCN, but the other two datasets are opposite. This means that on the UA12010, BlogCatelog and Flickr cosine graph is closer to the classification target than structure graph.
CONCLUSION
In this paper, aiming at the problem of semi-supervised graph node classification, we proposes novel dual graph complementary network (DGCN), which can utilize graphs that emphasize the different attributes of the input to guide the aggregation process. In addition, order to further capture richer information, we use two different branches to perform feature learning separately. At the same time, the disparity constraint is used between the two branches to further expand the difference. However, just using the diversity loss may retain too much unnecessary redundant infor- mation, which will interfere with the really important information. Therefore, our future work will try emphasize the common attributes in the embedding while expanding the differences between
8

---

Under review as a conference paper at ICLR 2021
The extensive experiments on several datasets further demonstrate the effectiveness of our DGCN algorithm.
branches.
In the future, will further study the correlation measurement of graphs and training objectives and further enrich our model with them.
REFERENCES
Sami Abu-El-Haija. Bryan Perozzi, Amol Kapoor, Nazanin Alipourfard, Kristina Lerman, Hrayr Harutyunyan, Greg Ver Steeg, and Aram Galstyan. Mixhop: Higher-order graph convolutional architectures via sparsified neighborhood mixing. arXiv preprint arXiv: iee:: 2019.
Thorsten Joachims. Transductive inference for text classification using support vector machines, In Ieml, volume 99, pp. 200-209, 1999.
Muhammad Raza Khan and Joshua E Blumenstock. Multi-gen: Graph convolutional networks for multi-view networks, with applications to global poverty. In Proceedings of the AAAI Conference Artificial Intelligence, volume 33, pp. 606-613, 2019.
Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional net- works. arXiv preprint arXiv: (w0.600999077. 2016.
Dong-Hyun Lee. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. Workshop on challenges in representation learning, ICML, volume 3, 2013.
Guohao Li, Matthias Muller, Ali Thabet, and Bernard Ghanem. Deepgens: Can gens go deep as cnns? Proceedings of the IEEE International Conference on Computer Vision, pp. 9267-9276, 2019.
Zaiqiao Meng, Shangsong Liang, Hongyan Bao, and Xiangliang Zhang. Co-embedding attributed networks. Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining, pp. 393-401, 2019.
Bryan Perozzi, Rami AI-Rfou, and Steven Skiena. Deepwalk: Online learning of social repre- sentations. In Proceedings of the 20th ACM SIGKDD international conference Knowledge discovery and data mining, pp. 701-710, 2014.
Salah Rifai, Pascal Vincent. Xavier Muller, Xavier Glorot, and Bengio. Contractive auto- encoders: Explicit invariance during feature extraction. 01 2011.
Jian Tang, Meng Qu, Mingzhe Wang, Ming Zhang, Jun Yan, and Qiaozhu Mei. Line: Large-scale information network embedding. In Proceedings of the 24th international conference world wide web, pp. 1067-1077, 2015.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pictro Lio, and Yoshua Bengio. Graph attention networks. arXiv preprint 17Xiv:1710.10003 2017.
Pascal Vincent, Hugo Larochelle, Bengio, and Pierre-Antoine Manzagol. Extracting and compos- ing robust features with denoising autoencoders. pp. 1096-1103, 01 2008. doi: 1390294.
Wenjun Wang, Xiao Liu, Pengfei Jiao, Xue Chen, and Di Jin. A unified weakly supervised frame- work for community detection and semantic matching. In Pacific-Asia Conference on Knowledge Discovery and Data Mining, pp. 218-230. Springer, 2018.
Xiao Wang, Houye Ji, Chuan Shi, Bai Wang, Yanfang Ye, Peng Cui, and Philip S Yu. Heterogeneous graph attention network. The World Wide Web Conference, pp. 2022-2032, 2019.
Xiao Wang, Meiqi Zhu, Deyu Bo, Peng Cui, Chuan Shi, and Jian Pei. Am-gen: Adaptive multi- channel graph convolutional networks. In Proceedings of the 26th ACM SIGKDD International Conference Knowledge Discovery Data Mining, pp. 1243-1253. 2020.

---

Under review as a conference paper at ICLR 2021
Jun Wu, Jingrui He, and Jiejun Xu. Net: Degree-specifer graph neural networks for node and graph classification. Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 406-415. 2019.
Zhilin Yang, William Cohen, and Ruslan Salakhudinov. Revisiting semi-supervised learning with graph embeddings. In International conference on machine learning, pp. 40-48. PMLR, 2016.
Xiaojin Zhu, Zoubin Ghahramani, and John Lafferty. Semi-supervised learning using gaussian fields and harmonic functions. volume pp. 912-919, 01 2003.
10


Reviews:
[{"review_id": "x8wVahzvXdE", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents a GNN model to jointly encode both topology and feature graphs to enhance node representations' quality. In particular, the model DGCN uses two GCNs to learn and propagate two different types of node representations on the topology graph, respectively. The model also utilizes two GATs to learn and propagate two different types of node representations on the feature graph, respectively. Finally, the model leverages attention mechanisms on these four types of node representations to produce the final node embeddings.\n\nPros: \n+ The model obtains promising results.\n\nCons:\n\n+ The motivation in the second paragraph of the introduction makes confusion. The quote sentence - \"Most of the traditional GNNs only consider the single connection between nodes and ignore other implicit information\" - leads to why not considering multiple connections between nodes as there are GNN works on hyper-graphs such as [2].\n+ The intuition in the third paragraph of the introduction is not clear as the paper does not have any ablation study for this intuition. \n\"Network performance is largely related to the quality of the graph, which usually emphasizes the relevance of an attribute of instances\", so which references for this intuition? What are the attributes of instances?\n+ Using v=1 for A_1 to denote the topology graph and v = 2 for A_2 to denote the feature graph makes the paper harder to read.\n+ The paper is not well written as it does not include any descriptions about model parameters in both the paper and the supplementary material. So it's hard to understand how to train DGCN and the baselines and how to analyze the model and ablation studies.\n+ The most important one is that, regarding the model architecture, DGCN is precisely similar to AM-GCN. In particular, DGCN changes from using two GCNs for the feature graph in AM-GCN [1] to using two GATs. Therefore, DGCN is straightforward and incremental (i.e., lacking novelty).\n\n[1] AM-GCN: Adaptive Multi-channel Graph Convolutional Networks. KDD 2020.\n\n[2] Hyper-SAGNN: a self-attention based graph neural network for hypergraphs. ICLR 2020.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "oQg1LB0ppPk", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces a method on semi-supervised graph classification. For each graph, the method first constructs another view based on the cosine similarity between nodes' features, and from the two views (topology and feature similarity), GCN and GAT are applied to extract representations. All node representations are further combined via two layers of attentions. A diversity loss that encourages dissimilarity between the learned representations of GCN and GAT is introduced to the cross-entropy loss for joint optimization. The whole framework makes sense in terms of learning meaningful node representations for classification. However, the method lacks novelty, it is an incremental development on the existing graph neural networks. The choice of GCN and GAT as the building blocks are not well justified. It is also possible to try other kinds of GNNs. The statement on GAT \"ignores the inherent structure of the graph space\" on page 4 is confusing since it learns weights based on the graph structures. The experimental results show the better performance of the proposed method, but are not well analyzed. It may be better to compare with other multi-graph methods such as\nYu Shi, Fangqiu Han, Xinwei He, Xinran He, Carl Yang, Jie Luo, and Jiawei Han. \"mvn2vec: Preservation and collaboration in multi-view network embedding.\" arXiv preprint arXiv:1801.06597 (2018).\nAlso, it seems AM-GCN in the experiments also works on multi-view of graphs. The superiority of the proposed method compared to AM-GCN is not clearly described in the paper.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "VAK0BPiSBO", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a dual complementary network framework for graph representation learning. Two graphs representing topology and features respectively are first constructed. Then, two branches leveraging the two graphs are proposed to explore different aspects of the original graph. Finally, a diversity loss is presented to capture the rich information of node features.\nTo me, the overall presentation is barely satisfactory, with many proposals not well-motivated. Also, the novelty is limited given the large body of existing work on exploring dual aspects of graphs. Moreover, the experiments are not convincing.\n\nDetailed comments:\n* The reason why two branches of GConv nets are used is not clear. I am especially not clear how the proposed DGCN differs from dual-channeled GAT, why the diversity loss is not employed on attention heads, and how does the embedding learnt by GCN supplement the information of that by GAT. More elaborations are needed.\n* Experiments are not convincing; the result analysis of this paper is rather superficial.\n  * Since DGCN uses two branches of GConv nets, large-scale datasets are necessary to evaluate the performance and efficiency.\n  * Inconsistency between GCN and kNN-GCN. It seems that on UAI2010, BlogCatelog, and Flickr kNN-GCN is significantly better than GCN, but the opposite holds for the other two datasets. It should be noted why the two methods show such different performance on different datasets.\n* Given the large amount of existing literature regarding dual networks, many related methods are missing. The authors should especially pay attention to network embedding techniques, e.g., [1].\n\nMinor:\n* Mathematical expressions are in chaotic forms, which makes the readability poor.\n* Page 5: CDAN -> DGCN?\n\n[1]\tZ. Meng, S. Liang, H. Bao, and X. Zhang, Co-Embedding Attributed Networks, in WSDM, 2019, pp. 393–401.", "extra_scores": {}, "extra_text": {}}, {"review_id": "8sriY-EOBYD", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work proposes a new method by combining GCN and GAT to perform node semi-supervised classification task. The new model uses node features to build another graph, uses GCN and GAT on the original graph and the new graph, and also adds a loss term to reduce the similarity of the final node representations.  \n\n1. The idea is very heuristic without much insight. The paper keeps arguing traditional GNNs only use one-side information, but they are actually leverage both node features and graph structures by propagating nodes features over the graph structure. So the statement is not correct. The paper claims that different node attributes contribute in different ways that should be sufficiently leveraged, but this is a very confusing argument if not paired with empirical justification. In the proposed model, it seems the authors try to resolve the above confusing issue via using another graph structure built only based on only node attributes. There is very unclear connection showing why this method resolves the problem they proposed. \n\n2. The experiment parts are not in a fair comparison too, as the paper does not use the standard way to perform dataset splitting. For this new splitting way, no hyperparameters are report for both the model here and previous models. Some benchmark datasets for semi-supervise learning are also not used, e.g., cora, pubmed,... \n\n3. There are quite a few errors in grammar. I suggest authors to perform some grammar checking. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All four reviewers expressed very significant and consistent concerns on this submission during review. No reviewer is willing to support this submission during discussion. It is clear this submission does not make the bar of ICLR.
Decision: Reject