Dual Graph Representation Learning

Abstract: Graph representation learning embeds nodes in large graphs as low-dimensional vectors and  benefit to many downstream applications. Most embedding frameworks, however, are inherently transductive and unable to  generalize to unseen nodes or learn representations across different graphs. Inductive approaches, such as GraphSAGE,  neglect different contexts of nodes and cannot learn node embeddings dually. In this paper, we present an unsupervised dual encoding framework, \textbf{CADE},  to generate context-aware representation of nodes by combining real-time neighborhood structure with neighbor-attentioned representation, and preserving extra memory of known nodes. Experimently, we exhibit that our approach is effective by comparing to state-of-the-art methods.

Reviews and author responses:

This paper proposed a dual graph representation method to learn the representation of nodes in a graph. In particular, it learns the embedding of paired nodes simultaneously for multiple times, and use the mean values as the final representation. The experimental result demonstrates some improvement over existing methods. Overall, the idea is presented clearly and the writing is well structured. But the novelty is limited. Specifically, 

1. The proposed method is very similar with the unsupervised GraphSAGE, which also optimizes Eq.(7). The  difference is that the proposed method learns a multi-channel representation and uses the attention technique to aggregate the multi-channel representation. Thus, the novelty is incremental.

2. Since the proposed method uses the multi-channel representation, how to set the number of channels $K$? How does this parameter affect the performance?

3. Some unsupervised network embedding baseline methods, such as DeepWalk and Node2Vec,  should be included into the experiment section. 



Strengths: 
1. This paper presents a unified framework CADE for unsupervised graph representation learning, which combining the local neighbors and global information. 
2. Based on GraphSAGE, the module DualENC can include positive pairs simultaneously, capturing the context-aware information.
3. In extensive experiments on Pubmed and PPI for link prediction, they demonstrate that their model beats the state-of-the-arts.

Drawbacks:
1. In this paper, the “Dual” refers to output adjacent nodes embeddings, where the adjacent nodes are collected by random walk. But the “dual” here is inconformity with its intrinsic definition and applications, according to the existing work “Dual-Primal Graph Convolutional Networks”. In other words, this paper lacks novelty, which seems incremental to the existing works. Specifically, the base-encoder is a variant of GraphSAGE plus a global embedding. The bi-attention is a variant of GAT.
2. The paper is not clear and structured. There are some confused logics and inaccurate interpretation. For example, in Introduction section, “Based on the hierarchical framework of GraphSAGE, GAT ….”. In Model section, there exists information redundancy and incoherence. For example, in section 3.2, equations 2-6 are reduplicative that have already appeared in algorithm 2.  Figure 1needs more information to help understand the model. To sum up, the connection and presentation of the four modules are not clear, which makes it difficult to read.
3. The experimental comparison methods and results are not complete. The baseline lacks the optimal method, “Deep Graph Infomax (DGI) ICLR 2019”, on unsupervised node classification. The results are not enough to support the claims (Only two benchmarks are compared).  Some important models on unsupervised graph representation learning are missing, e.g., Deep Graph Infomax(Veličković, 2018). 



This paper extends GraphSAGE in several dimensions: 1) applying attention when aggregating neighbors (already used by GAT and many other approached); 2) Ensembling node embedding by applying DualENC multiple times on positive pairs selected by random walk (this is doing aggregation of neighborhood again); and 3) adding global bias. All this makes the proposed method an incremental extension of existing solutions.  There is no theoretical justification why these extensions should work.

The evaluation only uses two baselines. However, there are many other inductive graph embedding learning approaches, such as (Hamilton et al. (2017b) Velickovic et al. (2017) Bojchevski & Gu ̈nnemann (2017) Derr et al. (2018) Gao et al. (2018), Li et al. (2018), Wang et al. (2018) and Ying et al. (2018b)). It is necessary to compare with these approaches. 


Meta-review: This work proposes context-aware representation of graph nodes leveraging attention over neighbors (as already done in previous work). Reviewers concerns about lack of novelty, lack of clarity of paper and lack of comparison to state of the art methods have not been addressed at all.
We recommend rejection.

Decision: reject