Title: DUAL GRAPH REPRESENTATION LEARNING

Abstract: Graph representation learning embeds nodes in large graphs as low-dimensional vectors and benefit to many downstream applications. Most embedding frameworks, however, are inherently transductive and unable to generalize to unseen nodes or learn representations across different graphs. Inductive approaches, such as GraphSAGE, neglect different contexts of nodes and cannot learn node embeddings dually. In this paper, we present a context-aware unsupervised dual encoding framework, CADE, to generate representation of nodes by combining real-time neighborhood structure with neighbor-attentioned representation, and preserving extra memory of known nodes. Experimently, we exhibit that our approach is effective by comparing to state-of-the-art methods.

1. INTRODUCTION:

The study of real world graphs, such as social network analysis (Hamilton et al. (2017a) ), molecule screening (Duvenaud et al. (2015) ), knowledge base reasoning (Trivedi et al. (2017) ), and biological protein-protein networks (Zitnik & Leskovec (2017) ), evolves with the development of computing technologies. Learning vector representations of graphs is effective for a variety of prediction and graph analysis tasks (Grover & Leskovec (2016) ; Tang et al. (2015) ). High-dimensional information about neighbors of nodes are represented by dense vectors, which can be fed to off-the-shelf approaches to tasks, such as node classification (Wang et al. (2017) ; Bhagat et al. (2011) ), link prediction (Perozzi et al. (2014) ; Wei et al. (2017) ), node clustering (Nie et al. (2017) ; Ding et al. (2001) ), recommender systems (Ying et al. (2018a) ) and visualization (Maaten & Hinton (2008) ).

There are mainly two types of models for graph representation learning. Transductive approaches (Perozzi et al. (2014) ; Grover & Leskovec (2016) ; Tang et al. (2015) ) are able to learn representations of existing nodes but unable to generalize to new nodes. However, in real-world evolving graphs such as social networks, new users will join and must be represented. Inductive approaches were proposed to address this issue. GraphSAGE (Hamilton et al. (2017b) ), a hierarchical sampling and aggregating framework, successfully leverages feature information to generate embeddings of the new nodes. However, GraphSAGE has its own faults. Firstly, it samples all neighborhood nodes randomly and uniformly; secondly, it treats the output of encoder as the final representation of node.

Based on the hierarchical framework of GraphSAGE, GAT (Velickovic et al. (2017) ) uses given class labels to guide attention over neighborhood so as to aggregate useful feature information. However, without knowledge of ground-truth class labels, it is difficult for unsupervised approaches to apply attention. To address this issue, we introduce a dual encoding framework for unsupervised inductive representation learning of graphs. Instead of learning self-attention over neighborhoods of nodes, we exploit the bi-attention between representations of two nodes that co-occur in a short random-walk (which we call a positive pair).

In Figure 1 , we illustrate how nodes are embedded into low-dimensional vectors, where each node v has an optimal embeddings o v . Yet the direct output of encoder z v of GraphSAGE could be located anywhere. Specifically, given feature input from both sides of a positive pair (v, v p ), a neural network is trained to encode the pair into K different embeddings z k v and z k vp through different sampled neighborhoods or different encoding functions. Then, a bi-attention layer is applied to generate the most adjacent matches z v|vp and z vp|v , which will be referred as dual-representations. By putting most attention on the pair of embeddings with smallest difference, dual representation of nodes with less deviation will be generated, which can be visualized as z v|• in Figure 1 . GraphSAGE naively assumes that unseen graph structure should be (easily) represented by known graphs data. We combine the ground truth structure and the learned dual-encoder to generate final representation. Unseen nodes can be represented based on their neighborhood structure. Current inductive approaches have no direct memory of the training nodes. We combine the idea of both transductive and inductive approaches via associating an additive global embedding bias to each node, which can be seen as a memorable global identification of each node in training sets.

Our contributions can be summarized as follows:

• we introduce a dual encoding framework to produce context-aware representation for nodes, and conduct experiments to demonstrate its efficiency and effectiveness; • we apply bi-attention mechanism for graph representation dual learning, managing to learn dual representation of nodes more precisely; • we combine the training of transductive global bias with inductive encoding process, as memory of nodes that are already used for training.

2. RELATED WORK:

Following (Cai et al. (2018) , Kinderkhedia (2019) and Goyal & Ferrara (2018) ), there are mainly two types of approaches:

2.1. NETWORK EMBEDDING:

For unsupervised embedding learning, DeepWalk (Perozzi et al. (2014) ) and node2vec (Grover & Leskovec (2016) ) are based on random-walks extending the Skip-Gram model; LINE (Tang et al. (2015) )seeks to preserve first-and second-order proximity and trains the embedding via negative sampling; SDNE (Wang et al. (2016) ) jointly uses unsupervised components to preserve secondorder proximity and expolit first-order proximity in its supervised components; TRIDNR (Pan et al. ( 2016)), CENE (Sun et al. (2016) ), TADW (Yang et al. (2015) ),GraphSAGE (Hamilton et al. (2017b) ) utilize node attributes and potentially node labels. Convolutional neural networks are also applied to graph-structured data. For instance, GCN (Kipf & Welling (2017) ) proposed an simplified graph convolutional network. These graph convolutional network based approaches are (semi-)supervised. Recently, inductive graph embedding learning (Hamilton et al. (2017b) 

2.2. ATTENTION:

Attention mechanism in neural processes have been extensively studied in neuroscience and computational neuroscience (Itti et al. (1998) ; Desimone & Duncan (1995) ) and frequently applied in deep learning for speech recognition (Chorowski et al. (2015) ), translation (Luong et al. (2015) ), question answering (Seo et al. (2016) ) and visual identification of objects (Xu et al. (2015) ). Inspired by (Seo et al. (2016) and Abu-El-Haija et al. ( 2018)), we construct a bi-attention layer upon aggregators to capture useful parts of the neighborhood.

3. MODEL:

Let G = {V, E, X} be an undirected graph, where a set of nodes V are connected by a set of edges E, and X ∈ R |V |×f is the attribute matrix of nodes. A global embedding bias matrix is denoted by

B ∈ R |V |×d

, where a row of B represents the d-dimensional global embedding bias of a node. The hierarchical layer number, the embedding output of the l-th layer and the final output embedding are denoted by L, h l and z, respectively.

3.1. CONTEXT-AWARE INDUCTIVE EMBEDDING ENCODING:

The embedding generation process is described in Algorithm 1. Assume that the dual encoder is trained and parameters are fixed.

Algorithm 1 Context-Aware Dual-Encoding (CADE) input: the whole graph G = (V, E); the feature matrix X; the trained DualENC output: learned embeddings z; 1: Run random walks on G to gain a set of positive pair P;

2: Z v ← ∅, ∀v ∈ V 3: for (v, v p ) ∈ P do 4: z v , z vp = DualENC(v, v p , G, X); 5: Z v ← Z v ∪ z v 6: Z vp ← Z vp ∪ z vp 7: end for 8: for v ∈ V do 9: z v = Mean(Z v ); 10: end for

After training, positive pairs are collected by random walks on the whole dataset. The features of each positive pair are passed through a dual-encoder. Embeddings of nodes are generated so that the components of a pair are related and adjacent to each other.

3.2. DUAL-ENCODER WITH MULTI-SAMPLING:

In this subsection, we explain the dual encoder.

Algorithm 2 DualENC input: Training graph G(V, E); node attributes X; global embedding bias matrix B; sampling times K; positive node pair (v, v p ); output: adjacent embeddings z v and z vp ;

1: For node v and v p , generate K embeddings, h v , h vp , using a base encoder SAGB 2: for i, j ∈ {1, ..., K} do 3:

S i,j ← α(h vi , h vpj ) 4: end for 5: softmax on flattened similarity matrix S: S i,j ← e S i,j ∑ K,K 0,0 e S i,j 6: calculate attention a v and a vp :

a vi ← ∑ K j=1 S i,j , a vpj ← ∑ K i=1 S i,j 7: z v ← ∑ K t=1 a vk h L vk 8: z vp ← ∑ K t=1 a vpk h L vpk

In the hierarchical sampling and aggregating framework (Hamilton et al. (2017b) ), it is challenging and vital to select relevant neighbor with the layer goes deeper and deeper. For example, given word "mouse" and its positive node "PC", it is better to sample "keyboard", instead of "cat", as a neighbor node. However, to sample the satisfying node according to heuristic rules layer by layer is very time consuming, and it is difficult to learn attention over neighborhood for unsupervised embedding learning.

As a matter of fact, these neighbor nodes are considered to be useful because they are more welcome to be sampled as input so as to produce more relevant output of the dual-encoder. Therefore, instead of physically sampling these neighbor nodes, in Step 2 to Step 6 in Algorithm 2, we directly apply a bi-attention layer on the two sets of embedding outputs with different sampled neighborhood feature as input, so as to locate the most relevant representation match, as a more efficient approach to exploring the most useful neighborhood.

We use the hierarchical sampling and aggregating framework as a base encoder in our experiments, but it can also be designed in many other ways.

Given node v, and node v p as a positive node pair, after sampling and aggregating for K times, we have K different representations h vk /h vpk corresponding to K different sampled neighborhoods, their similarity matrix can be calculated by

EQUATION

where α represents an dot-product operation.

our goal is to find the closest neighborhood match between v and v p within K × K possibilities, so we apply softmax on the flattened similarity matrix, and sum up by row(column). In this way, we manage to gain different attention over K neighborhoods of v(v p ) with respect to v p (v),

S ij ← exp(S ij ) ∑ K,K 0,0 exp(S ij )

(2)

a vi = K ∑ j=1 S ij

(3)

a vpj = K ∑ i=1 S ij (4)

and sum up the lth layer representations with attention as the dual-encoder outputs,

z v = K ∑ k=1 a vk h vk (5) z vp = K ∑ k=1 a vpk h vpk (6)

To train our encoder before using it to generate final representations of nodes, we apply a typical pair reconstruction loss function with negative sampling Hamilton et al. (2017b) :

EQUATION

where node v p co-occurs with v on fixed-length random walk (Perozzi et al. (2014) ), σ is the sigmoid function, P n is a negative sampling distribution, Q defines the number of negative samples. Note that z v and z vp are dual representation to each other while z vn represents the direct encoder output of negative sample v n .

3.3. DUAL-ENCODER WITH MULTI-AGGREGATING:

Besides learning dual representation with multiple sampling, we introduce another version of our dual encoder with multiple aggregator function.The intuition is that through different perspective, a node can be represented differently corresponding to different kinds of positive nodes.

In Step 1 in Algorithm 2, for a node v, we sample neighborhood once and aggregate feature with K sets of parameters, gaining K different representations h vk corresponding to K different character of v. Given a positive node pair, v and v p , their dual representation are calculated by applying biattention as we described in the last section. There is but one difference that we use a weigh vector A ∈ R 2d as parameter instead of dot-product, to calculate the K × K attention matrix between node v and node v p , As we can see in Figure 2 , to make sure that any embeddings of node v n as far away from any of node v as possible, it is equal to maximizing the distance between their support embeddings, which is the closest pair of embeddings of v and v n . The support embedding can be calculated by the learned dual encoder. In conclusion, our loss function can be modified as follows,

EQUATION

J G (z v ) = -log(σ(z T v z vp )) -Q • E vn∼Pn(v) log(σ(-z ′ T v z vn )) (9) z v , z vp = DualENC(v, v p , A) (10) z ′ v , z vn = DualENC(v, v p , A * ) (11)

where A * representing that we stop the back-propagation through A in dual encoding for negative sample node, since A are supposed to learn bi-attention between the positive node pair and be reused only to capture the support embedding of v and its negative sample nodes.

3.4. MEMORABLE GLOBAL BIAS IN HIERARCHICAL ENCODING:

In this section, we first explain the base encoder used in our proposed dual encoding framework, and then we introduce how we apply memorable global bias within this framework.

The general intuition of GraphSAGE is that at each iteration, nodes aggregate information from their local neighbors, and as this process iterates, nodes incrementally gather more and more information from further reaches of the graph. For generating embedding for one specific node u, we describe the process below. First, we construct a neighborhood tree with node u as the root, N u , by iteratively sampling immediate neighborhood of nodes of the last layer as children. Nodes at the lth layer are represented by symbol N l u , N 0 u = {u}. Then, at each iteration, each node i aggregates the representations of its children j, {h l-1 j }, and of itself, h l-1 i , into a single vector h l i , as representation of the next layer. After L iterations, we gain the Lth layer representation of v, as the final output.

While this framework generates good representation for nodes, it cannot preserve sufficient embedding informations for known nodes. More specifically, for nodes that are known but trained less than average, the learned model would have treated them like nodes unmet before. Therefore, we intuitively apply distinctive and trainable global bias to each node, as follows:

EQUATION

EQUATION

where B ∈ R |V |×d is the trainable global bias matrix, S(i) represents the sampled neighborhood and also the children nodes of node i in the neighborhood tree, AGGREGATE represents the neighborhood aggregator function, and || is a operator of concatenating vectors.

On one hand, B can be reused to produce embeddings for the known nodes or the unknown connected with the known, as supplement to the neural network encoder. On another hand, the global bias vectors can partially offset the uncertainty of the encoding brought by the random sampling not only during the training but also the final generation. Lastly but not least, we use only one set of global bias for all nodes, which means for any node, its representations of hidden layers are all added by the same bias vector. As a result of that, we are able to update parameters of aggregator function in the lowest layer with the global updated bias of nodes, highly increasing the training efficiency.

It is important for us to apply no global bias to the last layer of output, which is also the candidate of the dual-encoder output of nodes before applied with attention. The reason is that applying extra bias onto the last layer would directly change the embedding distribution of known nodes, making it unequal to the embedding distribution of unseen nodes. In general, the implementation of the base encoder with global bias is shown in Algorithm 3. The * in Step 8 means the children of node i in the neighborhood tree N u .

Algorithm 3 SAGB:sampling and aggregating with global bias input: node u; hierarchical depth L; weight matrices W l ; non-linearity σ; differentiable neighbor aggregator AGGREGATE l ; fixed-size uniform sampler

S : v → 2 V output: embedding z u ; 1: N 0 u = {u}; 2: for l = 1...L do 3: N l u ← {S(i), ∀i ∈ N l-1 u }; 4: end for 5: for l=1...L do 6: for i ∈ N 0 u ∪ N 1 u ∪ ... ∪ N L-l u do 7: h l-1 S * (i) ← AGGREGATE l ({h l-1 j , ∀j ∈ S * (i)}) 8: h l i ← σ(W l • [h l-1 i ||h l-1 S * (i) ]) 9: if l < L: h l i ← h l i + one hot(i) T B 10:

end for 11: end for 12: return z u ← h L u

4. EXPERIMENTS:

In this section, we compare CADE against two strong baselines in an inductive and unsupervised setting, on challenging benchmark tasks of node classification and link prediction. We also perform further studies of the proposed model in section 4.5.

4.1. DATASETS:

The following graph datasets are used in experiments and statistics are summarized in Table1:

• Pubmed: The PubMed Diabetes (Sen et al. ( 2008))1 dataset is a citation dataset which consists of scientific publications from Pubemd database pertaining to diabetes classified into one of three classes. Each publication in the dataset is described by a TF/IDF (Salton & Yu (1973) ) weighted word vector from a dictionary. • Blogcatalog: BlogCatalog2 is a social blog directory which manages bloggers and their blogs, where bloggers following each others forms the network dataset. • Reddit: Reddit3 is an internet forum where users can post or comment on any content. We use the exact dataset conducted by (Hamilton et al. (2017b) ), where each link connects two posts if the same user comments on both of them. • PPI: The protein-protein-interaction (PPI) networks dataset contains 24 graphs corresponding to different human tissues (Zitnik & Leskovec (2017) ). We use the preprocessed data also provided by (Hamilton et al. (2017b) ). We compare CADE against the following approaches in a fully unsupervised and inductive setting:

• GraphSAGE: In our proposed model, CADE, the base encoder mainly originates from GraphSAGE, a hierarchical neighbor sampling and aggregating encoder for inductive learning. Three alternative aggregators are used in Graphsage and CADE: (1) Mean aggregator, which simply takes the elementwise mean of the vectors in h k-1 u∈N (v) ; (2) LSTM aggregator, which adapts LSTMs to encode a random permutation of a node's neighbors' h k-1 ;

(3) Maxpool aggregator, which apply an elementwise maxpooling operation to aggregate information across the neighbor nodes.

• Graph2Gauss (Bojchevski & Günnemann (2017) ): Unlike GraphSAGE and my method, G2G only uses the attributes of nodes to learn their representations, with no need for link information. Here we compare against G2G to prove that certain trade-off between sampling granularity control and embedding effectiveness does exists in inductive learning scenario.

Beside the above two models, we also include experiment results of raw features as baselines. In comparison, we call the version of dual-encoder with multiple sampling as CADE-MS, while the version with multiple aggregator function as CADE-MA.

For CADE-MS, CADE-MA and GraphSAGE, we set the depth of hierarchical aggregating as L = 2, the neighbor sampling sizes as s 1 = 20, s 2 = 10, and the number of random-walks for each node as 100 and the walk length as 4. The sampling time in CADE-MS or the number of aggregator in CADE-MA is set as K = 10. And for all emedding learning models, the dimension of embeddings is set to 256, as for raw feature, we use all the dimensions. Our approach is impemented in Tensorflow (Abadi et al. (2016) ) and trained with the Adam optimizer (Kingma & Ba (2014) ) at an initial learning rate of 0.0001.

4.3. INDUCTIVE NODE CLASSIFICATION:

We evaluate the node classification performance of methods on the four datasets. On Reddit and PPI, we follow the same training/validation/testing split used in GraphSAGE. On Pubmed and Blogcatalog, we randomly selected 10%/20%/30% nodes for training while the rest remain unseen. We report the averaged results over 10 random split.

After spliting the graph dataset, the model is trained in an unsupervised manner, then the learnt model computes the embeddings for all nodes, a node classifier is trained with the embeddings of training nodes and finally the learnt classifier is evaluated with the learnt embeddings of the testing nodes, i.e the unseen nodes. Comparation on node classification performance on Pubmed and Blogcatalog dataset with respect to varying ratios of unseen nodes, are reported in Table 2 . CADE-MS and CADE-MA outperform other approaches on Pubmed. On Blogcatalog dataset, however, RawFeats performs best mainly because that, in Blogcatalog dataset, node features are not only directly extracted from a set of user-defined tags, but also are of very high dimensionality (up to 8,189) . Hence extra neighborhood information is not needed. As shown in Table 2 , CADE-MA performs better than CADE-MS, and both outperform GraphSAGE and G2G. CADE-MA is capable of reducing high dimensionality while losing less information than CADE-MS, CADE-MA is more likely to search for the best aggregator function that can focus on those important features of nodes. As a result, the 256-dimensional embedding learnt by CADE-MA shows the cloest node classification performance to the 8k-dimensional raw features.

Comparasion among GraphSAGE, CADE and other aggregator functions is reported in Figure3. Each dataset contains 30% unseen nodes. In general, the model CADE shows significant advance to the other two state-of-art embedding learning models in node classification on four different challenging graph datasets.

4.4. INDUCTIVE LINK PREDICTION:

Link prediction task evaluates how much network structural information is preserved by embeddings. We preform the following steps: (1) mark some nodes as unseen from the training of embedding learning models. For Pubmed 20% nodes are marked as unseen; (2) randomly hide certain percentage of edges and equal-number of non-edges as testing edge set for link prediction, and make sure not to produced any dangling node; (3) the rest of edges are then used to form the input graph for embedding learning and with equal number of non-edges form the training edge set for link predictor; (4) after training and inductively generation of embeddings, the training edge set and their corresponding embeddings will help to train a link predictor; (5) finally evaluate the performance on the testing edges by the area under the ROC curve (AUC) and the average precision (AP) scores. It implicates that searching for the best representation match through multiple sampling and biattention is efficient to filtering userful neighbor nodes without supervision from any node labels, and that the context-aware dual-encoding framework is capable of improving the inductive embedding learning ability without increasing sampling complexity. We aslo observe that CADE-gb, the variant simply adding the memorable global bias, continually shows advance in different sampling sizes.

4.5.2. THE NECESSARITY OF HIDDEN MEMORY:

It is necessary not to apply global bias to the encoder output. We compare the original framework GraphSAGE, CADE with the variants of our method: applying global bias only to the last layer, and only to the former layers. As Table 4 shows, CADE-gl performs poorly, while CADE-gb demonstrates the effect of keeping memory of the hidden representation for each node. 

5. CONCLUSION:

In this paper, we proposed CADE, an unsupervised and inductive network embedding approach which is capable of preserving local connectiveness distinctively , as well as learning and memorizing global identities for seen nodes while generalizing to unseen nodes. We applied a bi-attention architeture upon hierarchical aggregating layers to capture the most relevant representations dually for any positive pair. We also present an effective way of combining inductive and transductive ideas by allowing trainable global embedding bias to be retrieved in hidden layers within the hierarchical aggregating framework. Experiments demonstrate the superiority of CADE over the state-of-art baselines on unsupervised and inductive tasks. In the future, we would explore several possibilities, such as expanding dual encoding from pair-wise to n-wise, or using dual encoding framework in supervised embedding learning, or combing dual encoding with G2G by learning distribution representations dually for positive pairs.

:

Available at https://linqs.soe.ucsc.edu/data.

http://www.blogcatalog.com/

http://www.reddit.com/

PPI is a multi-label dataset.

Figures and Tables:

Figure 1: Visual comparison between representations learnt by current methods and dual encoding.

Figure 2: training support embeddings of node v and its negative sample node vn.

Figure 3: Classification results (micro-averaged F1 scores) w.r.t different aggregators on four datasets

Figure 4: node classification performance (micro-f1 score) w.r.t varying sampling sizes

Dataset Statistics

Table content: <table><tr><td>Dataset</td><td>Nodes</td><td>Edges</td><td colspan="3">Classes Features Avg Degree</td></tr><tr><td>Pubmed</td><td>19717</td><td>44324</td><td>3</td><td>500</td><td>4.47</td></tr><tr><td>Blogcatalog</td><td>5196</td><td>171743</td><td>6</td><td>8189</td><td>66.11</td></tr><tr><td>Reddit</td><td colspan="2">232,965 11,606,919</td><td>41</td><td>602</td><td>100.30</td></tr><tr><td>PPI</td><td>56944</td><td>818716</td><td>121 4</td><td>50</td><td>28.76</td></tr><tr><td colspan="2">4.2 EXPERIMENTAL SETTINGS</td><td/><td/><td/><td/></tr></table>

Prediction results for Pubmed/Blogcatalog w.r.t different unseen ratio

Table content: <table><tr><td>Methods</td><td/><td>Pubmed</td><td/><td/><td>Blogcatalog</td><td/></tr><tr><td>unseen-ratio</td><td>10%</td><td>30%</td><td>50%</td><td>10%</td><td>30%</td><td>50%</td></tr><tr><td>RawFeats</td><td colspan="6">79.22 77.66 77.74 90.00 89.05 87.08</td></tr><tr><td>G2G</td><td colspan="6">80.70 76.67 76.31 62.35 56.19 48.46</td></tr><tr><td colspan="7">GraphSAGE 82.05 81.32 79.68 71.48 69.33 64.92</td></tr><tr><td>CADE-MS</td><td colspan="6">84.25 83.40 81.74 77.35 73.71 70.88</td></tr><tr><td colspan="7">CADE-MA 84.56 83.03 82.40 84.33 82.21 79.04</td></tr></table>

Link prediction results for Pubmed/PPI w.r.t different percentage of hidden-edges .60 MS requires multiple neighborhood sampling, which increases the complexity of embedding learning. Yet by comparing CADE-MS against GraphSAGE with the same quantity of sampled neighborhood per node, the superiority of CADE model over existing models is still vast. In practice, we set the sampling layer L = 2 and the first-layer sampling size as 20. For the second layer, denote by s ′ 2 the sampling size in GraphSAGE, and by s 2 and T the sampling size and sampling time in CADE-MS. We compare the two methods with s ′ 2 = s 2 * K. A variant of CADE, called CADE-gb, applies only memorable global bias and no dual-encoding framework,

Table content: <table><tr><td>Dataset</td><td>Methods</td><td colspan="2">90%:10%</td><td colspan="2">80%:20%</td><td colspan="2">60%:40%</td><td>40%:60%</td></tr><tr><td/><td/><td>AUC</td><td>AP</td><td>AUC</td><td>AP</td><td>AUC</td><td>AP</td><td>AUC</td><td>AP</td></tr><tr><td/><td>RawFeats</td><td colspan="7">57.61 54.72 58.51 56.19 54.47 52.82 52.41 50.77</td></tr><tr><td colspan="9">Pubmed 64.13 68PPI G2G RawFeats 57.46 56.99 57.34 56.86 57.35 56.75 56.83 56.36 G2G 60.62 58.98 60.99 59.38 61.05 59.54 60.93 59.49 GraphSAGE 82.74 81.20 82.21 80.66 82.11 80.51 82.07 80.70</td></tr><tr><td/><td colspan="8">CADE-MS 85.87 85.08 84.21 83.48 84.46 82.84 83.61 82.39</td></tr><tr><td/><td colspan="8">CADE-MA 86.33 85.32 85.85 84.63 84.15 82.15 81.98 79.54</td></tr><tr><td colspan="9">Comparation on performance with respect to varying percentage of hidden edges are reported in</td></tr><tr><td colspan="7">Table3. CADE shows best link prediction performance on both datasets.</td><td/></tr><tr><td colspan="2">4.5 MODEL STUDY</td><td/><td/><td/><td/><td/><td/></tr><tr><td colspan="5">4.5.1 SAMPLING COMPLEXITY IN CADE-MS</td><td/><td/><td/></tr><tr><td colspan="2">Our proposed CADE-</td><td/><td/><td/><td/><td/><td/></tr></table>

node classification performance (f1-micro score) with different way of applying global bias

Table content: <table><tr><td>method</td><td>F1-micro</td></tr><tr><td>GraphSAGE</td><td>50.22</td></tr><tr><td>CADE-gl</td><td>39.24</td></tr><tr><td>CADE-gb</td><td>54.13</td></tr><tr><td>CADE</td><td>58.22</td></tr></table>

Peer review (iclr2020):
Strengths: 1. This paper presents a unified framework CADE for unsupervised graph representation learning, which combining the local neighbors and global information. 2. Based on GraphSAGE, the module DualENC can include positive pairs simultaneously, capturing the context-aware information. 3. In extensive experiments on Pubmed and PPI for link prediction, they demonstrate that their model beats the state-of-the-arts. Drawbacks: 1. In this paper, the “Dual” refers to output adjacent nodes embeddings, where the adjacent nodes are collected by random walk. But the “dual” here is inconformity with its intrinsic definition and applications, according to the existing work “Dual-Primal Graph Convolutional Networks”. In other words, this paper lacks novelty, which seems incremental to the existing works. Specifically, the base-encoder is a variant of GraphSAGE plus a global embedding. The bi-attention is a variant of GAT. 2. The paper is not clear and structured. There are some confused logics and inaccurate interpretation. For example, in Introduction section, “Based on the hierarchical framework of GraphSAGE, GAT ….”. In Model section, there exists information redundancy and incoherence. For example, in section 3.2, equations 2-6 are reduplicative that have already appeared in algorithm 2. Figure 1needs more information to help understand the model. To sum up, the connection and presentation of the four modules are not clear, which makes it difficult to read. 3. The experimental comparison methods and results are not complete. The baseline lacks the optimal method, “Deep Graph Infomax (DGI) ICLR 2019”, on unsupervised node classification. The results are not enough to support the claims (Only two benchmarks are compared). Some important models on unsupervised graph representation learning are missing, e.g., Deep Graph Infomax(Veli?kovi?, 2018).