Unsupervised Universal Self-Attention Network for Graph Classification
Authors: Dai Quoc Nguyen
Tu Dinh Nguyen
Dinh Phung
Venue:  (2020)
Keywords: Graph embedding
graph classification
universal self-attention network
graph neural network

Abstract: Existing graph embedding models often have weaknesses in exploiting graph structure similarities, potential dependencies among nodes and global network properties. To this end, we present U2GAN, a novel unsupervised model leveraging on the strength of the recently introduced universal self-attention network (Dehghani et al., 2019), to learn low-dimensional embeddings of graphs which can be used for graph classification. In particular, given an input graph, U2GAN first applies a self-attention computation, which is then followed by a recurrent transition to iteratively memorize its attention on vector representations of each node and its neighbors across each iteration. Thus, U2GAN can address the weaknesses in the existing models in order to produce plausible node embeddings whose sum is the final embedding of the whole graph. Experimental results show that our unsupervised U2GAN produces new state-of-the-art performances on a range of well-known benchmark datasets for the graph classification task. It even outperforms supervised methods in most of benchmark cases.

Under review as a conference paper at ICLR 2020
A UNIVERSAL SELF-ATTENTION GRAPH NEURAL NETWORK
Anonymous authors
Paper under double-blind review
ABSTRACT
Existing graph embedding models often have weaknesses in exploiting graph structure similarities, potential dependencies among nodes and global network properties. To this end, we present U2GNN, a novel embedding model leveraging on the strength of the recently introduced universal self-attention network (De- lighnni et al. 2019). to learn low-dimensional embeddings of graphs which can be used for graph classification. In particular, given an input graph, U2GNN first ap- plies self-attention computation, which is then followed by a recurrent transition to iteratively memorize its attention on vector representations its neighbors across each iteration. Thus, U2GNN can address the weaknesses in the existing models in order to produce plausible node embeddings whose sum is the final embedding of the whole graph. Experimental results in both supervised and unsupervised fashions show that our U2GNN produces new state-of of--hh--rtttt performances on range of well-known benchmark datasets for the graph classi- fication task.
each node and
INTRODUCTION
Many real-world and scientific data are represented in forms of graphs, e.g. data from knowledge graphs, recommender systems, social and citation networks well as telecommunication and bi- ological networks (Battaglia et al., 20188) Zhang et al.] 2018c). In general. graph can be viewed as a network of nodes and edges, where nodes correspond to individual objects and edges encode relationships among those objects. For example, in online forum, each discussion thread can be con- structed as graph where nodes represent users and edges represent commenting activities between users (Yanardag & Vishwanathan 2015).
Early approaches focus on computing the similarities among graphs to build graph kernel for graph classification (Gärtner et al. 2003 Kashima et al. 2003 Borgwardt & Kriegel, 2005 Shervashidze ct al. [2099] Vishwanathan et al. al.] [2010] SeerosShheasshzzze et al. 201 Yanardag & Vishwanathan [2015] Narayanan al.) [017 Ivanov & Burnaev, 2018). These graph kernel-based approaches treat each atomic substructure (e.g., subtree structure, random walk or shortest path) as an individual feature, and count their frequencies construct numerical vector to represent the entire graph, hence they ignore node attributes, substructure similarities and global network properties.
One recent notable strand is to learn dineeinnsional continuous embeddings of the whole graphs (aamiloon et al. 2017b; Zhang et al. 2018a Zhou et al. (2018), and then use these learned embed- predict graph labels (Wu et al. (019).. Advanced approaches in this direction have attempted to exploit graph neural network (Scarselli et al. 2009). capsule network
dings to train a classifier
(Sabour al. (017) or graph convolutional neural network (Kipf Welling [017 Hamilton al.,
2017a) for supervised learning objectives (Li et al. 20166] NNcperrt et al.
2016: Zhang et al. 2018b
Ying et al. 2018. Verma & Zhang, 2018: Xu et al. [2019] [inni & Chen 2019; Maron et al. 2019b
Chen et al. 2019). These graph neural network (GNN)-based approaches usually consist of two common phases: the propagating phase and the readout phase. The former phase aims to iteratively update vector representation of each node by recursively aggregating representations of its neigh- bors, and then the latter phase applies a pooling function (e.g., mean, max or sum pooling) on output node representations
produce an embedding of each entire graph; and this graph embedding is used to predict the graph label. We find that these approaches are currently showing very promis- ing performances, nonetheless the dependency aspect among nodes, which often exhibit strongly

---

Under review as a conference paper at ICLR 2020
in many kinds of real-world networks, has not been exploited effectively due to lack of advanced computations within the propagating phase.
Very recently, the universal self-attention network (Dehghami al. 2019) has been shown to be very powerful in NLP tasks such as question answering, machine translation and language model- ing. Inspired by this new attention network, we propose U2GNN novel universal self-attention graph neural network to learn plausible node and graph embeddings. Our intuition comes from an observation that the recurrent attention process in the universal self-attention network can memorize implicit dependencies between each node and its neighbors from previous iterations, which can be then aggregated to further capture the dependencies among substructures into latent representations in subsequent iterations; this process, hence, can capture both local and global graph structures. Al- gorithmically, each timestep, our proposed U2GNN iteratively exchanges a node representation with its neighborhood representations using a self-attention mechanism (Vaswani et al., 2017) fol- lowed by recurrent transition to infer node embeddings. We finally take the sum of all learned node embeddings to obtain the embedding of the whole graph. Our main contributions are as follows:
In our proposed U2GNN, the novelty of memorizing the dependencies among nodes implies that U2GNN can explore the graph structure similarities locally and globally an important feature that most of existing approaches are unable to do.
U2GNN can be seen as a general framework where we prove the powerfulness of our model in both supervised or unsupervised fashions. The experimental results on well-known benchmark datasets for the graph classification task show that both our supervised and unsupervised U2GNN models produce new state-of-the- (SOTA) accuracies in most of benchmark cases.
To our best of knowledge, our work is the first show that unsupervised model can noticeably outperform up-to-date supervised approaches by a large margin. Therefore, we suggest that future GNN works should pay more attention to the unsupervised fashion as well as not comparing supervised models with unsupervised models together. This is important in both industry and academic applications in reality where expanding unsupervised GNN models is more suitable due to the limited availability of class labels.
RELATED WORK
Early popular approaches are based on "graph kernel" which aims recursively decompose each graph into "atomic substructures" (e.g., graphlets, subtree structures, random walks or shortest paths) in order to measure the similarity between two graphs (Gärtner et al. 2003). For this reason, we can view each atomic substructure as a word token and each graph as text document, hence we represent collection of graphs as a document-term matrix which describes the normalized fre- quency of terms in documents. Then, we use dot product to compute the similarities among graphs to derive a kernel matrix used to measure the classification performance using a kernel- learning algorithm such as Support Vector Machines (SVM) (Hofmann et al. 2008). We refer to an overview of the graph kernel-based approaches in (Nikolentzos et al. 2019 Kriege et al. 2019).
Since the introduction of word emhedding models i.e., Word2Vec (Mikolov al. 2013) and Doc2Vec (Le & Mikolov] 2014), there have been several efforts attempted to apply them for the graph classification task. Deep Graph Kernel (DGK) (Yanardag & Vishwanathan 2015) applies Word2Vec to learn embeddings of atomic substructures to create the kernel matrix. Graph2 Vec Narayanan al. 2017) employs Doc2Vec to obtain embeddings of entire graphs in order to train
SVM classifier to perform classification. Anonymous Walk Embedding (AWE) (Ivanov & Burnaev 2018 maps random walks into "anonymous walks" which are considered as word tokens, and then utilizes Vec to achieve the graph embeddings to produce the kernel matrix.
In parallel, another recent line of work has focused on using deep neural networks to perform the graph classification in supervised manner. PATCHY-SAN (Niepert et al. 2016) adapts graph la- beling procedure to generate a fixed-length sequence of nodes from an input graph, and orders k-hop neighbors for each node in the generated sequence according to their graph labelings; PATCHY-SAN then selects a fixed number of ordered neighbors for each node and applies a convolutional neural network to classify the input graph. MPNN (Gilmer al., 2017). DGCNN (Zhang et al. 2018b) and DuFFPooL (Ying et 2018) are end-to-ent supervised models which share similar two-phase process by (i) using stacked multiple graph convolutional layers (e.g., GCN layer (Kipf & Welling
2

---

Under review as a conference paper at ICLR 2020
GraphSAGE layer (Hamilton et al., 2017a)) to aggregate node feature vectors, and (ii) ap- plying a graph-level pooling layer (c.g., mean, max or sum pooling, sort pooling or differentiable pooling) obtain the graph embeddings which are then fed to a fully-connected layer followed by a softmax layer to predict the graph labels.
2017)
Graph neural networks (GNNs) (Scarselli et al. 2009) aim to iteratively update the vector represen- tation of each node by recursively propagating the representations of its neighbors using a recurrent function until convergence. The recurrent function can be neural network e.g., gated recurrent unit (GRU) (Li et al. 2016), or multi-layer perceptron (MLP) (Xu et al. 2019). Note that both stacked GCN and GraphSAGE multiple layers can be seen as variants of the recurrent function in GNNs. Other graph embedding models are briefly summarized in (Zhou et al. 2018 Zhang et al. 2018c
Wu et al. 2019).
THE PROPOSED U2GNN
In this section, we detail how construct our U2GNN and then present how U2GNN learns model parameters to produce node and graph emheddings.
Leamning node embeddings
learning
graph
Irauition Funchion
Tsieps
Summing node embeddings
Feature Vector
Graph embedding
Figure Illustration of our U2GNN learning process, e.g., for node 3 with d 2 and N
Graph classification. Given a set of graphs "2.,, 9N3 C G and their corresponding class labels {y1, yN. S V. our U2GNN aims to learn plausible embedding og of each entire graph G in order to predict its label
Each graph is defined as G EV, where V is set of nodes, is of edges, and X R represents feature vectors of nodes. In U2GNN, as illustrated in Figure [ we universal self-attention network (Dehghani et al. (201)) to learn a node embedding of each node e V, and then og is simply returned by summing all learned node embeddings as follows
0g SUM (fo,, Vv V2)
Constructing U2GNN. Formally, given an input graph G, we uniformly sample of N neighbors for each v E V, and then use node v and its neighbors for the U2GNN learning process example, as illustrated Figure D we generate set of N neighbors {1, 2, 4) for node and then consider {3, 1, 2, 4} as an input to U2GNN where we leverage on the universal self-attention network (Dehghani al. 2019) learn an effective embedding of node
? For
Intuitively, the universal self-attention network can help to better aggregate feature vectors from neighbors of a given node to produce its plausible embedding. In particular, each node and its neighbors is transformed into sequence of feature vectors which are then iteratively refined at each timestep- - using a self-attention mechanism /Vaswani et al. 2017) followed by recurrent transition (TRANS) with adding residual connection (He et al. 2016 and layer normalization (LNORM) (Ba et al. 2016).
Given node v V. consider sequence of (N 1) nodes (v where v1 and {vr (v N+1
N sampled neighbors of We obtain input sequence of feature vectors (h) J== for which
Thee experimental results in Xu ct al. 2019) show that the sum pooling performs better than the mean and max poolings.
We sample different of neighhors at each training step.
3

---

Under review as a conference paper at ICLR 2020
h" X,, R In U2GNN, each step t, we feed (1a li=1 an input sequence and produce an output sequence (hi, Ji= for which hy, E IRd as follows:
N+1
h = LNORM (x,, + TRANS (x.)) with X LNORM (h%,
(2)
(1-1 ATT (h(-1)
(3)
where TRANS(.) and ()) denote feed-forward network and self-attention network respec- tively as follows:
TRANS (xi.) W2ReLU (Wixi, + bbi b
(4)
where W1 Rixdd and W2 E Rd are weight matrices, and b and b bias parameters, and:
N-1 Oii (vht,")
ATT (h 1)
where E Rddd is peoje-projcttion weight matrix; Oz,j is an attention weight which is computed using the softmax function over scaled dot products between the i-th and j-th input nodes:
(Qh
(Khh (-1
ai softmax
(6)
Vd
where Q E Raxd and K R are query-projection and key-projection matrices, respectively
Algorithm 1: The unsupervised learning process.
Algorithm 2: The supervised learning process.
Imput: G EV) for do
Input: G (V,E,X) with its label y for v do
SAMPLE (v:) N+11 where [v:) are N sampled neighbors of Vi =V
N+1
SAMPLE [v. M++1 where [ are N sampled neighhors of V1 =V for = 1, 2, do vi E (1, 2,.,..... (N +1)
1N+1
r= 1, 2..,, 7 do
Vi E (1,2, (N 1))
x + LNORM (h.
ATT (h)
x < LNORM (h. (-1
ATT (h
h, + LNORM (x), TRANS (x.))
h. - INORM (xL. TRANS (x())
0, h2, VVEV 0g SUM((O, WEE
og SUM
y Wo,
Learning parameters of U2GNN in "unsupervised" fashion: After T steps, we use the vector representation h" to infer node embeddings Ov, For example, as shown Figure we have = 3, V2 1, = 2 and Va 4, and then consider h to infer We learn our model parameters (including the weight matrices and biases as well node embeddings o,) by minimizing the sampled softmax loss function (Jean et al. 2015) applied to node E V as follows:
exp(o, h
LUZGNN (v) log Ever exp(o, - h²"
where V is a subset sampled from V.
We briefly describe the general learning process of our proposed U2GNN model in Algorithm I Here, the learned node embeddings o, are used as the final representations of nodes v E V. After that, obtain the plausible embedding og of the graph by summing all learned node embeddings as mentioned in Equation [ Finally, we employ logistic regression classifier from LIBLINEAR
(Fan al. 2008) predict the graph labels.
'It important note that GAT Vellčkoviic et al. 2018 borrows the standard attention technique from (Bahdanau et al. 2015) in using single-layer feedforward neural network parametrized by weight vector and then applying the LeakyReLU non lineliiyrity function followed by the softmax function to compute importance weights of neighbors of given node. Therefore, GAT is much different with the self-attention mechanism. More information about GCN, GraphSAGF: and GAT can be found in Appendix

---

Under review as a conference paper at ICLR 2020
Learning parameters of U2GNN in "supervised" fashion: We do not need to learn the node embeddings separately, hence after T steps, use hi as the final embedding of node vi V (i.e., o, = h,") We then feed Og to fully connected layer to predict the graph labels as briefly presented in Algorithm 2 Finally, we learn the model parameters by minimizing the cross entropy loss function.
In general, on the node level, each node and its neighbors are iteratively attended in the recur- rent process with weight matrices shared across timesteps and iterations, thus U2GNN can memorize the potential dependencies among nodes within substructures. On the graph level, U2GNN views the shared weight matrices as memories to access the updated node-level information from previ- ous iterations further aggregate broader dependencies among substructures into implicit graph representations in subsequent iterations. Therefore, U2GNN is advantageous to capture both global and local graph structures to learn effective node and graph embeddings, leading to state-of-the-ar performances for the graph classification task.
Proof:
EXPERIMENTAL SETUP
We prove the effectiveness of U2GNN the graph classification task using range of well- known benchmark datasets for both supervised and unsupervised fashions.
4.1 DATASETS
We use well-known datasets consisting of 5 social network datasets (COLLAB, IMDB-B, BB, IMDB- M. RDT-B and RDT-M5K) (Yanardag & Vishwanathan (013)) and 4 bioinformatics datasets (DD, MUTAG, PROTEINS and PTC). We follow (Niepert et 2016 Zhang et al. (0188) to use node degrees features all social network datasets these datasets do not have available node fea- tures. Table reports the statistics these datasets.
Social networks datasets. COLLAB is scientific dataset where each graph repre- sents collaboration network of corre- sponding researcher with other researchers from each of physics fields; and each graph is labeled to physics field the researcher be- long to. IMDB-B and IMDB-M are movie collaboration datasets where each graph is derived from actor/actress and genre infor- mation of different movies IMDB, in which nodes correspond to actors/actresses, and each edge represents co-appearance of two actors/actresses in the same movie; and each graph is assigned a genre. RDT- B and RDT-M5K are datasets derived from Reddit community, in which each online dis- cussion thread is viewed as a graph where nodes correspond linked if at least one of them replied to another's comment; and each graph la- beled to sub-community the corresponding
Dataset
#G
#Cls
A.NG
A.NN
d
COLLAB
5,000
3
74.5
65.9
IMDB-B
1,000
19.8
IMDB-M
1,500
3
13.0
10.1
RDT-B
2,000
2
429.6
2.3
RDT-M5K
5,000
508.5
2.3
DD
1,178
284.3
82
MUTAG
188
2
17.9
2.2
7
PROTEINS
1,113
2
39.1
3.7
3
PTC
344
2
25.6
2.0
19
Table 1: Statistics the experimental benchmark datasets. #G denotes the numbers of graphs. #CIs denotes the number of graph classes. A.NG denotes the average number of nodes per graph. A.NN de- notes the average number of neighbors per node. d is the dimension of node feature vectors (i.e. the num- ber of node labels).
users, two
thread belongs to.
Bioinformatics datasets. DD (Dobson & Doig 2003) is collection of 1,178 protein network struc- tures with 82 discrete node labels, where each graph is classified into enzyme or non-enzyme class. PROTEINS comprises 1,113 graphs obtained from (Borgwardt et al. 2005) to present secondary 188 nitro compound
structure elements (SSEs). MUTAG (Debnath et al. 1991) is a collection networks with 7 discrete node labels, where classes indicate mutagenic effect bacterium. PTC (Toivonen et al. 2003) consists of 344 chemical compound networks with 19 discrete node labels where classes show carcinogenicity for male and female rats.
5

---

Under review as a conference paper at ICLR 2020
4.2 TRAINING PROTOCOL TO LEARN GRAPH EMBEDDINGS
Coordinate embedding: The relative coordination among nodes might provide meaningful infor- mation about graph structure. We follow Dehghani al. (2019) associate each position at step I a pre-defined coordinate embedding p using the sinusoidal functions (Vaswani et al. 2017), thus
we can change Equation in Section to:
x LNORM (h., + ATT (h + p.))
(8)
with Phps sin((/100003//4) + si(t/100000///))
P1,2+1 = cos(i/100003)//) + cox(/100000))//
From the preliminary experiments, adding coordinate embeddings enhances classification results MUTAG and PROTEINS in the unsupervised fashion, hence we use the coordinate embeddings only for these datasets.
Hyper-parameter setting to learn our model parameters for all experimental datasets: (i) Re- garding the unsupervised fashion, fix the hidden size of the feed-forward network in Equation
4 to 1024 (k 1024). and the number of samples in the sampled loss function LU2GNN to 512 TV' 512) in Equation 7 We set the batch size 512 for COLLAB, DD and RDT-B: 1024 for RDT-M5K; and 128 for remaining datasets. (ii) Regarding the supervised fashion, we vary the the hidden size in {32, 128, 512, 1024) and the batch size {1, 2. 4). (iii) For both the unsupervised and supervised fashions, and the number T steps from {1, 2. 3, 4, 5, 63. We apply the Adam optimizer (Kingma & Ba 2014) to train U2GNN model and apply grid search to select the Adam initial learning rate Ir E (5e -5 le 4 5e le 31. We run up to 50 epochs and evaluate the model as in what follows.
the number N of neighbors sampled for each node from {4, 8, 16)
4.3 EVALUATION PROTOCOL
For each dataset, after obtaining the graph embeddings, we perform the same evaluation process from (Yanardag & Vishwanathan, 2015 Niepert et al. 2016 Zhang et al.) 2018b Xu et al. 2019 Xinyi & Chen 2019), which is using 10-fold cross- validation scheme calculate the classificttion performance for a fair comparison
Baseline models: We compare our U2GNN with up-to-date strong baselines as follows:
Unsupervised approaches: Graphlet Kernll (GK) (Shervashidze et al.
2009). Weisfeiler-
Lehman kernel (WL) (Shervashidze et al. 201T), Deep Graph Kernel (DGK) Yanardag & Vish-]
wanathan 2015) and Anonymous Walk Embedding (AWE) (Ivanov & Burnaev (018)).
Supervised approaches: PATCHY-SAN (PSCN) (Niepert et al. 2016). Graph Convolutional
Network (GCN) (Kipf & Welling 2017). GraphSAGE (Hamilton al., 2017a). Deep Graph
CNN (DGCNN) (Zhang et al. (018b). Graph Capsule Convolution Neural Network (GCAPS)
Verma & Zhang (0118). Capsule Graph Neural Network (CapsGNN) (Xinyi & Chen 2019)
Graph Isomorphism Network (GIN) (Xu et 2019), Graph Feature Network (GFN) et al.,
2019), Invariant Equivariinn Graph Network (IEGN) (Maron et al. 2019b), Provably Powerful
Graph Network (PPGN) (Maron et al. 2019a) and Discriminative Structural Graph Classification
(DSGC) (Seo et al. 2019).
We report the baseline results taken from the original papers published in [Ivanov & Burnaev
2018
Verma & Zhang 2018 Xinyi & Chen 2019 Chen et al. 2019 Seo et al. 2019).
EXPERIMENTAL RESULTS
Table presents the experimental results on the 9 benchmark datasets. We use "Unsupervised" to denote the unsupervised graph embedding models that can access all nodes from the whole dataset, but not the graph labels; and we use "Supervised" to denote the supervised models that use the graph labels of training graphs during training.
RReaaddin the unsupervised fashion, we the logistic regression classifier from L.BLI (Fan et al. 2008
with setting the termination criterion 0.001.

---

Under review as a conference paper at ICLR 2020
In general, unsupervised U2GNN achieves state-of-the-an performances range of bench- marks for the graph classification task, hence this demonstrates a high impact of the unsupervised U2GNN in inferring the plausible node and graph embeddings. It is note that several existing works (Xinyi & Chen 2019, Xu et al.) 2019; Chen et al., 2019; Maron et al. 2019h) [Soo et al, 2019) compared the supervised models with the unsupervised models together due to the of the same 10-fold cross-validation scheme. However, our unsupervised U2GNN significantly out- performs the supervised methods most of benchmark cases by a large margin of 18+ %, thus suggest that future GNN works should separate two training fashions.
Model
COLLAB
IMDB-B
IMDR-M
RDT-B
RDT-M5K
GK 2009
72.84 0.28
65.87 0.98
43.89 0.38
77.34 + 0.18
41.01 0.17
Unsuperviser
WL 2011
79.02 + 1.77
73.40 + 4.63
49.33 +.33+-47 4.75
81.10
49.44 + 2.36
DGK 2015
73.09 + 0.25
66.96 + 0.56
44.55
78.04 + 0.39
41.27 + 0.18
AWE (2018
73.93 + 1.94
74.45 + 5.83
51.54
87.89 + 2.53
50.46 1.91
U2GNN
95.62 0.92
93.50 2.27
74.80 4.11
84.80 + 1.53
77.25 1.46
DSGC 2019
79.20 +: 1.60
73.20 -: 4.90
48.50 1 4.80
92.20 + 2.40
GFN 2019
81.50 + 2.42
73.00 + 4.35
51.80 + 5.16
57.59 + 2.40
PPGN 2
81.38 + 1.42
73.00 + 5.77
50.46 + 3.59
Supervised
GIN (2019)
80.20 1.90
75.10 5.10
52.30 2.80
92.40 2.50
$7.50 1.50
IEGN 2019b
77.92 + 1.70
71.27 4.50
48.55 3.90
CapsGNN (2019)
79.62 + 0.91
73.10 4.83
50.27 2.65
50.46 1.91
GCAPS 2018
77.71 + 2.51
71.69 + 3.40
48.50 4.10
87.61 + 2.51
50.10 1.72
DGCNN (2018b)
73.76 0.49
70.03 + 0.86
0.85
76.02 + 1.73
48.70
GCN (2017)
81.72 -: 1.64
73.30 1: 5.29
51.20 5.13
56.81 2.37
GraphSAGk 2017a
79.70 + 1.70
72.40 + 3.60
49.90 + 5.00
89.10 + 1.90
PSCN 2016
72.60 + 2.15
71.00 + 2.29
45.23 + 2.84
86.30 + 1.58
49.10 0.70
U2GNN
77.84 1.48
79.40 + 4.35
56.20 3.35
80.25 + 2.38
50.90
Model
DD
PROTEINS
MUTAG
PTC
(2009
78.45 + 0.26
71.67 0.55
81.58 2.11
57.26
Unsupervised
WL 2011
79.78 1 0.36
74.68 1 0.49
82.05 = 0.36
57.97 + 0.49
DGK 2015
73.50 + 1.01
75.68 1 0.54
87.44 = 2.72
60.08 + 2.55
AWE 20018
71.51 +: 4.02
87.87 = 9.76
U2GNN
95.67 1.89
78.07 3.36
81.34 6.56
84.59 5.12
DSGC 2019
77.40 1 6.40
74.20 + 3.80
86.70 = 7.60
GFN 2019
78.78 + 3.49
76.46 1 4.06
90.84 = 7.22
PPGN 2019a
77.20 + 4.73
90.55 = 8.70
66.17 1: 6.54
Supervised
GIN (2019
76.20 2.80
89.40
64.60 + 7.00
IEGN 201
75.19 1999:44 4.30
84.61 10.0
59.47 = 7.30
CapsGNN 2019
75.38 4.17
76.28 + 3.63
86.67 = 6.88
GCAPS 2018
77.62 4.99
76.40:1..1
66.01
DGCNN 2018b
79.37 0.94
$5.54:0.99
85.83 = 1.66
58.59 +: 2.47
GCN 2011
75.65 + 3.24
87.20 = 5.11
GraphSAGE 20
65.80 4.90
65.90 + 2.70
79.80 = 13.90
PSCN 2016
77.12 2.41
75.89 + 2.76
92.63 4.21
62.29 5.68
U2GNN
81.24 + 1.84
78.53 + 4.07
89.97 + 3.65
79.36 + 4.06
Table 2: Graph classification results (% accuracy). The best scores are in bold.
Regarding our supervised U2GNN, on the social network datasets, our model produces new state- of-the-art performances on IMDB-B and IMDB-M, especially U2GNN significantly achieves 4+% absolute higher accuracies than all supervised baselines. In addition, U2GNN obtains promising scores on COLLAB, RDT-B and RDT-M5K. On the bioinformatics datasets, our U2GNN obtains new highest accuracies DD. PROTEINS and PTC. Moreover, U2GNN produces competitive accuracy compared with those of baseline models on MUTAG, for which there are no significant differences between supervised U2GNN and some supervised baselines (e.g., GFN, GIN and

---

Under review as a conference paper at ICLR 2020
PPGN). Note that there only 188 graphs in the MUTAG dataset, which explains the high variance in the results.
N234
11.3
N1MM
Effects the number of steps (I') (in the 3 top figures), and the number of neighbors (N) sampled for each node (in the 3 bottom figures) in the unsupervised fashion. Regarding each dataset, for all 10 folds, we vary the value of either or N while using the same fixed values of other sypee--arraeetes
Figure 2:
Next we investigate the effects of hyper-parametens on the experimental datasets in the unsupervised fashion in Figure 2 In general, our unsupervised U2GNN could consistently obtain better results than those of baselines with any value of T and N, as long as the training process is stopped precisely for all datasets. boosts the performance on bioinformatics data. A possible reason is that the bioinformatics datasets comprise sparse networks where the average number of neighbors per node below as shown in Table [T] hence we need to use more steps to learn graph properties. In addition, using small N generally produces higher performances on the bioinformatics datasets, while using higher values of N is more suitable for the social network datasets. Besides, the social network datasets are much denser than the bioinformatics datasets, thus this is reason why we should use more sampled neighbors the social networks rather than the bioinformaties ones. Note that the similar findings also occur in the supervised fashion.
particular, we find that higher T helps on most the datasets, and especially
Knde embedding
Graph embedding
40
-390
-100
-200
-208
Figure 3: visualization the node and graph embeddings learned by our unsupervised U2GNN on the DD dataset. We do not include the embedding visualization of other baselines because those methods are significantly different from our unsupervised U2GNN.
To qualitatively demonstrate the effectiveness of capturing the local and global graph properties, we use t-SNE (Maaten & Hinton 2008) to visualize the learned node and graph embeddings in the
Mooe figures he found in Appendix B
8

---

Under review as a conference paper at ICLR 2020
unsupervised fashion the DD dataset where the node labels are available. It can be seen from Figure 3 that our unsupervised U2GNN can effectively capture the local structure wherein the nodes clustered according to the node labels, and the global structure wherein the graph embeddings well-separated from each other; verifying the plausibility of the learned node and graph embeddings.
6 CONCLUSION
In this paper, we introduce a novel graph neural network model U2GNN for the graph classification task. Given an input graph, U2GNN applies self-attention mechanism followed by a recurrent transition to learn the node embeddings and then sums all the learned node embeddings to obtain the embedding of the entire graph. Our U2GNN achieves new highest accuracies on most of well-known benchmark datasets in both the supervised and unsupervised fashions, using the same 10-fold cross-validation scheme, against the up-to-date unsupervised and supervised baselines. In future works, we plan to investigate the effectiveness of U2GNN on other important tasks such as node classification and link prediction. Our code is available at: https://nnonymous-uri/
REFERENCES
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey Hinton. Layer normalization. arXiv: 1607.06450, 2016.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning align and translate. ICLR, 2015.
Peter W Battaglia, Jessica B Hamrick, Victor Bapst, Alvaro Sanchez-Gonzalez, Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro. Ryan Faulkner, et al. Relational inductive biases, deep learning, and graph networks. 1xxv::80660012/1 2018.
Karsten M. Borgwardt and Hans-Peter Kriegel. Shortest-Path Kernels on Graphs. In ICDM, pp- 74-81, 2005.
Karsten M Borgwardt, Cheng Soon Ong, Stefan Schönauer, SVN Vishwanathan, Alex Smola, and Hans-Peter Kriegel. Protein function prediction via graph kerelss. Bioinformatics, (s(upp). i47-i56. 2005.
Ting Chen, Song Bian, and Yizhou Sun. Are powerful graph neural nets necessary? a dissection on graph classification. arXiv: 2019.
Asim Kumar Debnath, Rosa Lopez de Compadre, Gargi Debnath, Alan Shusterman, and Cor- win Hansch. Structure-activity relationship of mutagenic aromatic and heteroaromatic nitro com- pounds. Correlation with molecular orbital energies and hydrophobicity. Journal of medicinal chemistry, 34(2):78--777, 1991.
Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser. Universal Transformers. ICLR, 2019.
Paul D Dobson and Andrew J Doig. Distinguishing enzyme structures from non-enzymes without alignments. Journal of molecular biology, 330(4);771-783, 2003.
Rong-En En Fan, Kai-Wei Wei Chang, Cho-Jui Hsieh, Kiang-Rui Wang, and Chih-Jen Lin. LIBLINEAR: A Library for Large Linear Classification. Journal of Machine Learning Research, 9:1871-1874, 2008.
Thomas Gärtner, Peter Flach, and Stefan Wrobel. On graph kernels: Hardness results and efficient alternatives. In Learning Theory and Kernel Machines, pp. 129-143. 2003.
Justin Gilmer, Samuel S. Schoenholz, Patrick F. Riley, Oriol Vinyals, and George E. Dahl. Neural Message Passing for Quantum Chemistry. ICML, pp. 1263-1272 2017.
William L. Hamilton, Rex Ying, and Jure Leskovec. Inductive representation learning on large graphs. In NIPS, pp. 1024-1034, 2017a.

---

Under review as a conference paper at ICLR 2020
William L. Hamilton, Rex Ying, and Jure Leskovec. Representation learning graphs: Methods and applications. arXiv: 2017b.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, pp. 770-778, 2016.
Thomas Hofmann, Bernhard Schölkopf, and Alexander J Smola. Kernel methods in machine learn- ing. The annals of statistics, pp. 1171-1220, 2008.
Sergey Ivanov and Exgeny Burnaev. Anonymous walk embeddings. In ICMI,, pp. 2191-2200, 2018.
Sébastien Jean, Kyunghyun Cho, Roland Memisevic, and Yoshua Bengio. On using very large target vocabulary for neural machine translation. In ACL, pp. 1-10, 2015.
Hisashi Kashima, Koji Tsuda, and Akihiro Inokuchi. Marginalized kernels between labeled graphs. In ICML, pp. 321-328, 2003.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv:1412.6988. 2014.
Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional net- works. In ICLR, 2017.
Nils M Kriege, Fredrik D Johansson, and Christopher Morris. A survey on graph kernels. 1903.19033113 2019.
Quoe Le and Tomas Mikolov. Distributed representations of sentences and documents. ICML, pp. 1188-11811 119,, 2014.
Yujia Li, Daniel Tarlow, Mare Brockschmidt, and Richard Zemel. Gated Graph Sequence Neural Networks. ICLR, 2016.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 2-579-2605, 2008.
Haggai Maron, Heli Ban-Hamu, Hadar Serviansky, and Yaron Lipman. Provably powerful graph networks. arXiv: 11.../1111 1166111161221111111111111 2019a.
Haggai Maron, Heli Ban-Hamu, Nadav Shamir, and Yaron Lipman. Invariant and equivariant graph networks. ICLR, 2019b.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. Distributed representations of words and phrases and their compositionality. In NIPS, pp. 3111-3119, 2013.
Annamalai Narayanan, Mahinthan Chandramohan, Rajasekar Venkatesan, Lihui Chen, Yang Liu, and Shantanu Jaiswal. graph2vec: Learning distributed representations of graphs. 17X1v:1707.05005. 2017.
Mathias Niepert, Mohamed Ahmed, and Konstantin Kutzkov. Learning Convolutional Neural Net- works for Graphs. In ICMI., pp. 2014-2023, 2016.
Giannis Nikolentzos, Giamnis Siglidis, and Michalis Vazirgiannis. arXiv:1904.122 122188 2019.
Graph kernels: A survey.
Bryan Perozzi, Rami AI-Rfou, and Steven Skiena. DeepWalk: Online Learning of Social Represen- tations. In SIGKDD, pp. 701-710, 2014.
Sara Sabour, Nicholas Frosst, and Geoffrey E Hinton. Dynamic routing between capsules. NIPS, pp. 3859-3869, 2017.
Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. The graph neural network model. IEEE Transactions on Neural Nenworks, 20( 10:1100,,,, 2009.
Younjoo Seo, Andreas Loukas, and Nathanael Peraudin. Discriminative structural graph classifica- tion. arXiv: /905. 13422, 2019.
10

---

Under review as a conference paper at ICLR 2020
Nino Shervashidze, SVN Vishwanathan, Tobias Petri, Kurt Mehlhorn, and Karsten Borgwardt. Ef- ficient Graphlet Kernels for Large Graph Comparison. In AISTATS, pp. 488 0888-95, 2009.
Nino Shervashidze, Pascal Schweitzer, Erik Jan van Leeuwen, Kurt Mehlhorn, and Karsten M Borg- wardt. Weisfeiler-lhmaan graph kernels. Journal of Machine Learning Research, 12(Sep):2539- 2561, 2011.
Hannu Toivonen, Ashwin Srinivasan, Ross D King, Stefan Kramer, and Christoph Helma. Statistical evaluation of the predictive toxicology challenge 2000-2001. Bioinformatics, 19(10):1133-119) 2003.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. NIPS, pp. 5998-6008, 2017.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph Attention Networks. ICLR, 2018.
Saurabh Verma and Zhi-Li Zhang. Graph capsule convolutional neural networks. arXiv: 2018.
S Vichy N Vishwanathan, Nicol N Schraudolph, Risi Kondor, and Karsten M Borgwardt. Graph kernels. Journal of Machine Learning Research, 120A1))11011122,, 2010.
Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and Philip S Yu. A comprehensive survey on graph neural networks. arXiv: TrXXv:/901.00596 2019.
Zhang Xinyi and Lihui Chen. Capsule Graph Neural Network. ICLR, 2019.
Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. Networks? ICLR, 2019.
How Powerful Are Graph Neural
Pinar Yanardag and SVN Vishwanathan. Deep graph kernels. In SIGKDD, pp. 1365-1374, 2015.
Rex Ying, Jiaxuan You, Christopher Morris, Xiang Ren, William L. Hamilton, and Jure Leskovec. Hierarchical graph representation learning with differentiable pooling. In NeurIPS, pp. 4805- 4815, 2018.
Daokun Zhang, Jie Yin, Xingquan Zhu, and Chengqi Zhang. Network representation learning: A survey. IEEE Transactions on Big Data, to appear, 2018a.
Muhan Zhang, Zhicheng Cui, Marion Neumann, and Yixin Chen. An End-to-End Deep Learning Architecture for Graph Classification. In AAAI, 2018b.
Ziwei Zhang, Peng Cui, and Wenwu Zhu. Deep learning on graphs: A survey, arXiv:1812.04202, 2018c.
Jie Zhou, Ganqu Cui, Zhengyan Zhang, Cheng Yang, Zhiyuan Liu, and Maosong Sun. Graph neural networks: A review of methods and applications. 1XXiv:812200434 2018.
A GNN VARIANTS
Regarding GCN Kipp & Welling 2017). each node v E V has a feature vector m, E ,Rd,,and and GCN infers new vector representation h, dd for node from its neighbors follows:
h g
Σ (Wa,, +b) VVVV DEN.UEV)
where W E Rddxdd and b E dR are a weight matrix and a bias respectively, is a non-linear activation function, and NG is the of neighbors of node v. GCN can indirectly capture nodes many hops away by using multiple GCN layers stacked on top of each other as follows:
h (++1) =g
Σ (UENOUYV
(w(K) + )(()) .VVE
(10)
11

---

Under review as a conference paper at ICLR 2020
where A is the layer index, and h. Bu- The vector outputs of the last GCN layer used the final vector representations of nodes. GCN can be jointly trained with classification classifier by minimizing the cross-entropy loss function.
GraphSAGE (Hamilton et al. 2017a) is an extension of GCN, which the representation h.(++1 of node after k-th GraphSAGE layer is produced as follows:
h, (k+1)
( SAGE
h
N. "I) vv V
(11)
NN, can be computed in several ways where [+] denotes vector concatenation operation, and (k+1) such as using the GCN computation as follows:
(1+1) h NG
= g Σ
(wrl Gex + b(()) WWEE
(12)
WEN,
or applying an element-wisse max-pooling operation as follows:
(k+1) max {Lg thoooth!* (k) + b) Yu eu
(13)
N. can be computed by taking the mean (or sum) of all vectors in (hu Vu E M. NG is defined fixed-size, uniformly drawn from the of all neighbor nodes of v, and uniformly sampled differently through each layer. The vector outputs of the last GraphSAGE layer are used infer node embeddings. GraphSAGE employs the random walk algorithm (Perozzi et al.) 2014) to generate context nodes for each target node and then adapts the negative sampling technique (Mikolov et al. 2013) to learn the model parameters.
In addition, h 1)
GAT Veličković et al. 2018) extends GCN in assigning importance weights to neighbors of a given node by exploring the standard attention technique (Bahdanau et al., 2015). GAT induces a new vector representation h, of node from its neighbors as follows:
h 8
Σ
Tv.u Way
VVEV
(14)
(E)
where Tu, is an importance weight which is computed as follows:
exp (LeakyReLU (a [a, @ Wz,l))
Tv.u EveN, exp (LeakyReLU (a [Wa, x, @ W W)rl))
(15)
B EFFECTS OF HYPER PARAMETERS
12

---

Under review as a conference paper at ICLR 2020
Figure 4: Effects of the number of steps (T)) in the unsupervised fashion. Regarding each dataset, for all 10 folds, we vary the value while using the same fixed values of other
PREEET
Figure 5: Effects of the number of neighbors (N) sampled for each node in the unsupervised fashion. Regarding each dataset, for all 10 folds, we vary the value while using the same fixed values of other paramtearameters.
13


Reviews:
[{"review_id": "HyxQAQLatr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors developed a graph embedding method called U2GAN based on self-attention mechanism. Similar to many existing graph neural network, U2GAN samples and aggregate neighboring features for each node in a graph. The aggregation function is similar to GAT, i.e., a query based attention layer. The difference is an incorporation of a transition function followed the attention layer. By minimizing Eq. 7, node embeddings can be inferred, which are summed up to obtain a graph embedding, for the downstream graph classification task.\n\nThere are several points that are unclear in the paper.\n1.  The major argument of the advantage of using self-attention for neighborhood aggregation is to facilitate memorizing the dependencies between nodes and explore the graph structure similarities locally and globally. This argument, however, was not clearly discussed in the paper. First, it is not clear on why existing GNN methods, such as GCN, GraphSage, and GAT, cannot do so. Second, it is not clear on how does the proposed U2GAN achieve it. The current paper only provides some high-level descriptions. A more specific or theoretical discussion is desired.\n2. Since the attention based aggregation is similar to GAT, a discussion on the difference is important.\n3. Several model designs are not well justified. In Eq. 2, 3, the reason to employ Layernorm is missing. In Eq. 7, the intuition on how does the loss function help learn effective embeddings remains to be clarified. Also, it may be better to evaluate different pooling method to obtain graph embedding to justify the choice of sum in Eq. 1.\n4. Since the proposed method aims to learn node embeddings in an unsupervised manner, it is better to see some descriptions on why graph classification was selected as the task in evaluation, instead of node classification.\n5. In the experiments, some methods such as deepwalk, node2vec, graphsage and GAT are missing in comparison. In particular, due to the similarity between the proposed method GAT, it is interesting to evaluate GAT by replacing its supervised loss by Eq. 7 as a compared method. Moreover, in fig. 4, other visualizations of other methods can be compared to demonstrate the difference between the proposed method and others.\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "Hkl1LHjCYS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The submission proposes a graph neural network based on propagation with the attention mechanism. Then the output function uses the summation of node vectors to read out information about the graph. \n\n While the design is good, all components are all known techniques: the sampling procedure is like GraphSAGE; the propagation rule is similar to GAT, and the output function is wide uses in graph neural networks. \n\nCritics: \n\nThe writing is not clear. At the top of page 4: quote: \"... and produce an output sequence {h_vi^t}i=1^N+1\". Do you keep only the vector h_v1 and throw away other vectors? Because you will also put v_2 at the center and compute its vector in a different self-attention computation. If this is the case, why not just say the output is h_v1? If this is not the case, then each node will have multiple vectors: one is computed when the node is at the center, and others are computed when the node is sampled as a neighbor. \n\nBelow Boris Knyazev has several comments, which are not well addressed by authors. There is a discussion about transductive learning and inductive learning. However, it seems the authors still don't know how to run inductive learning on the graph classification task (quote \"... still do not have a standard inductive setting for the graph classification task where we only use a part of each graph during training.\"). Boris does not suggest to use part of each graph; instead, he suggests not using test graphs. I believe this is the standard practice in inductive learning (e.g. kernel methods). \n\nAnother comment from Boris about the case when T=1, and the response is \"T=1 does not correspond to a single layer network\". I don't understand the response either. When T=1, a node only gets information from its neighbors. It is similar to a one-layer GCN or GAT, in which a node also only gets information from its immediate neighbors. \n\nI also don't understand why the author insists that the proposed model has a layer-based architecture. In my view, it is a graph neural network by the standard of propagation rule and output function. \n\n\n\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "BkltXbVRKB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n\n\nThe paper presents an new unsupervised model for graph classification. It borrows the idea from universal self-attention network and applies it to graph learning. It achieves surprisingly good results on benchmark datasets. Despite the good results, I do not think the technical quality is good enough to make it accepted. My concerns contain the following aspects:\n\n1.\tIf we compare the proposed model with the graph attention networks (GAT), it just adds the recurrent transition and the layer normalizer, which are also from the universal self-attention. This makes the paper not novel enough.  Furthermore, adding these components are not so related to unsupervised learning, it does not add any value to the unsupervised learning strategy.\n2.\tThe description of the unsupervised learning objective is not clear. From Algorithm 1, it seems $o_v$ is equal to $h_v^T$, I cannot understand the meaning of Eq. (7) at all.\n3.\tThe results are too good to be true. Although we cannot judge it based on this belief, the authors have to convince the readers and explain how the huge performance gain is obtained (on some datasets U2GAN is even 27% higher than all of other methods).  I understand the experimental setting is transductive, but even that cannot explain everything. To justify the experiments, the authors need to do a lot of ablation study, such as comparing with supervised learning version of this model, while in the paper there is no ablation model to explain it.\n\n\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: All three reviewers are consistently negative on this paper. Thus a reject is recommended.
Decision: Reject