Introduction
When performing network embedding, one maps network nodes into vector representations that reside in a low-dimensional latent space. Such techniques seek to encode topological information of the network into the embedding, such as affinity BIBREF0 , local interactions (e.g, local neighborhoods) BIBREF1 , and high-level properties such as community structure BIBREF2 . Relative to classical network-representation learning schemes BIBREF3 , network embeddings provide a more fine-grained representation that can be easily repurposed for other downstream applications (e.g., node classification, link prediction, content recommendation and anomaly detection).
For real-world networks, one naturally may have access to rich side information about each node. Of particular interest are textual networks, where the side information comes in the form of natural language sequences BIBREF4 . For example, user profiles or their online posts on social networks (e.g., Facebook, Twitter), and documents in citation networks (e.g., Cora, arXiv). The integration of text information promises to significantly improve embeddings derived solely from the noisy, sparse edge representations BIBREF5 .
Recent work has started to explore the joint embedding of network nodes and the associated text for abstracting more informative representations. BIBREF5 reformulated DeepWalk embedding as a matrix factorization problem, and fused text-embedding into the solution, while BIBREF6 augmented the network with documents as auxiliary nodes. Apart from direct embedding of the text content, one can first model the topics of the associated text BIBREF7 and then supply the predicted labels to facilitate embedding BIBREF8 .
Many important downstream applications of network embeddings are context-dependent, since a static vector representation of the nodes adapts to the changing context less effectively BIBREF9 . For example, the interactions between social network users are context-dependent (e.g., family, work, interests), and contextualized user profiling can promote the specificity of recommendation systems. This motivates context-aware embedding techniques, such as CANE BIBREF9 , where the vector embedding dynamically depends on the context. For textual networks, the associated texts are natural candidates for context. CANE introduced a simple mutual attention weighting mechanism to derive context-aware dynamic embeddings for link prediction. Following the CANE setup, WANE BIBREF10 further improved the contextualized embedding, by considering fine-grained text alignment.
Despite the promising results reported thus far, we identify three major limitations of existing context-aware network embedding solutions. First, mutual (or cross) attentions are computed from pairwise similarities between local text embeddings (word/phrase matching), whereas global sequence-level modeling is known to be more favorable across a wide range of NLP tasks BIBREF11 , BIBREF12 , BIBREF13 , BIBREF14 . Second, related to the above point, low-level affinity scores are directly used as mutual attention without considering any high-level parsing. Such an over-simplified operation denies desirable features, such as noise suppression and relational inference BIBREF15 , thereby compromising model performance. Third, mutual attention based on common similarity measures (e.g., cosine similarity) typically yields dense attention matrices, while psychological and computational evidence suggests a sparse attention mechanism functions more effectively BIBREF16 , BIBREF17 . Thus such naive similarity-based approaches can be suboptimal, since they are more likely to incorporate irrelevant word/phrase matching.
This work represents an attempt to improve context-aware textual network embedding, by addressing the above issues. Our contributions include: ( INLINEFORM0 ) We present a principled and more-general formulation of the network embedding problem, under reproducing kernel Hilbert spaces (RKHS) learning; this formulation clarifies aspects of the existing literature and provides a flexible framework for future extensions. ( INLINEFORM1 ) A novel global sequence-level matching scheme is proposed, based on optimal transport, which matches key concepts between text sequences in a sparse attentive manner. ( INLINEFORM2 ) We develop a high-level attention-parsing mechanism that operates on top of low-level attention, which is capable of capturing long-term interactions and allows relational inference for better contextualization. We term our model Global Attention Network Embedding (GANE). To validate the effectiveness of GANE, we benchmarked our models against state-of-the-art counterparts on multiple datasets. Our models consistently outperform competing methods.
Problem setup
We introduce basic notation and definitions used in this work.
Proposed Method
Model framework overview
To capture both the topological information (network structure INLINEFORM0 ) and the semantic information (text content INLINEFORM1 ) in the textual network embedding, we explicitly model two types of embeddings for each node INLINEFORM2 : ( INLINEFORM3 ) the topological embedding INLINEFORM4 , and ( INLINEFORM5 ) the semantic embedding INLINEFORM6 . The final embedding is constructed by concatenating the topological and semantic embeddings, i.e., INLINEFORM7 . We consider the topological embedding INLINEFORM8 as a static property of the node, fixed regardless of the context. On the other hand, the semantic embedding INLINEFORM9 dynamically depends on the context, which is the focus of this study.
Motivated by the work of BIBREF9 , we consider the following probabilistic objective to train the network embeddings:
DISPLAYFORM0
where INLINEFORM0 represents sampled edges from the network and INLINEFORM1 is the collection of model parameters. The edge loss INLINEFORM2 is given by the cross entropy DISPLAYFORM0
where INLINEFORM0 denotes the conditional likelihood of observing a (weighted) link between nodes INLINEFORM1 and INLINEFORM2 , with the latter serving as the context. More specifically,
DISPLAYFORM0
where INLINEFORM0 is the normalizing constant and INLINEFORM1 is an inner product operation, to be defined momentarily. Note here we have suppressed the dependency on INLINEFORM2 to simplify notation.
To capture both the topological and semantic information, along with their interactions, we propose to use the following decomposition for our inner product term:
DISPLAYFORM0
Here we use INLINEFORM0 to denote the inner product evaluation between the two feature embeddings INLINEFORM1 and INLINEFORM2 , which can be defined by a semi-positive-definite kernel function INLINEFORM3 BIBREF26 , e.g., Euclidean kernel, Gaussian RBF, IMQ kernel, etc. Note that for INLINEFORM4 , INLINEFORM5 and INLINEFORM6 do not reside on the same feature space. As such, embeddings are first mapped to the same feature space for inner product evaluation. In this study, we use the Euclidean kernel
INLINEFORM0
for inner product evaluation with INLINEFORM0 , and linear mapping
INLINEFORM0
for feature space realignment with INLINEFORM0 . Here INLINEFORM1 is a trainable parameter, and throughout this paper we omit the bias terms in linear maps to avoid notational clutter.
Note that our solution differs from existing network-embedding models in that: ( INLINEFORM0 ) our objective is a principled likelihood loss, while prior works heuristically combine the losses of four different models BIBREF9 , which may fail to capture the non-trivial interactions between the fixed and dynamic embeddings; and ( INLINEFORM1 ) we present a formal derivation of network embedding in a reproducing kernel Hilbert space.
Direct optimization of ( EQREF9 ) requires summing over all nodes in the network, which can be computationally infeasible for large-scale networks. To alleviate this issue, we consider other more computationally efficient surrogate objectives. In particular, we adopt the negative sampling approach BIBREF27 , which replaces the bottleneck Softmax with a more tractable approximation given by
DISPLAYFORM0
where INLINEFORM0 is the sigmoid function, and INLINEFORM1 is a noise distribution over the nodes. Negative sampling can be considered as a special variant of noise contrastive estimation BIBREF28 , which seeks to recover the ground-truth likelihood by contrasting data samples with noise samples, thereby bypassing the need to compute the normalizing constant. As the number of noise samples INLINEFORM2 goes to infinity, this approximation becomes exact BIBREF29 . Following the practice of BIBREF27 , we set our noise distribution to INLINEFORM3 , where INLINEFORM4 denotes the out-degree of node INLINEFORM5 .
We argue that a key to the context-aware network embedding is the design of an effective attention mechanism, which cross-matches the relevant content between the node's associated text and the context. Over-simplified dot-product attention limits the potential of existing textual network embedding schemes. In the following sections, we present two novel, efficient attention designs that fulfill the desiderata listed in our Introduction. Our discussion follows the setup used in CANE BIBREF9 and WANE BIBREF10 , where the text from the interacting node is used as the context. Generalization to other forms of context is straightforward.
Optimal-transport-based matching
We first consider reformulating content matching as an optimal transport problem, and then re-purpose the transport plan as our attention score to aggregate context-dependent information. More specifically, we see a node's text and context as two (discrete) distributions over the content space. Related content will be matched in the sense that they yield a higher weight in the optimal transport plan INLINEFORM0 . The following two properties make the optimal transport plan more appealing for use as attention score. ( INLINEFORM1 ) Sparsity: when solved exactly, INLINEFORM2 is a sparse matrix with at most INLINEFORM3 non-zero elements, where INLINEFORM4 is the number of contents ( BIBREF30 , INLINEFORM5 ); ( INLINEFORM6 ) Self-normalized: row-sum and column-sum equal the respective marginal distributions.
Implementation-wise, we first feed embedded text sequence INLINEFORM0 and context sequence INLINEFORM1 into our OT solver to compute the OT plan,
DISPLAYFORM0
Note that here we treat pre-embedded sequence INLINEFORM0 as INLINEFORM1 point masses in the feature space, each with weight INLINEFORM2 , and similarly for INLINEFORM3 . Next we “transport” the semantic content from context INLINEFORM4 according to the estimated OT plan with matrix multiplication
DISPLAYFORM0
where we have treated INLINEFORM0 as a INLINEFORM1 matrix. Intuitively, this operation aligns the context with the target text sequence via averaging the context semantic embeddings with respect to the OT plan for each content element in INLINEFORM2 . To finalize the contextualized embedding, we aggregate the information from both INLINEFORM3 and the aligned INLINEFORM4 with an operator INLINEFORM5 ,
DISPLAYFORM0
In this case, we practice the following simple aggregation strategy: first concatenate INLINEFORM0 and the aligned INLINEFORM1 along the feature dimension, and then take max-pooling along the temporal dimension to reduce the feature vector into a INLINEFORM2 vector, followed by a linear mapping to project the embedding vector to the desired dimensionality.
Attention parsing
Direct application of attention scores based on a low-level similarity-based matching criteria (e.g., dot-product attention) can be problematic in a number of ways: ( INLINEFORM0 ) low-level attention scores can be noisy (i.e., spurious matchings), and ( INLINEFORM1 ) similarity-matching does not allow relational inference. To better understand these points, consider the following cases. For ( INLINEFORM2 ), if the sequence embeddings used do not explicitly address the syntactic structure of the text, a relatively dense attention score matrix can be expected. For ( INLINEFORM3 ), consider the case when the context is a query, and the matching appears as a cue in the node's text data; then the information needed is actually in the vicinity rather than the exact matching location (e.g., shifted a few steps ahead). Inspired by the work of BIBREF31 , we propose a new mechanism called attention parsing to address the aforementioned issues.
As the name suggests, attention parsing re-calibrates the raw low-level attention scores to better integrate the information. To this end, we conceptually treat the raw attention matrix INLINEFORM0 as a two-dimensional image and apply convolutional filters to it:
DISPLAYFORM0
where INLINEFORM0 denotes the filter banks with INLINEFORM1 and INLINEFORM2 respectively as window sizes and channel number. We can stack more convolutional layers, break sequence embedding dimensions to allow multi-group (channel) low-level attention as input, or introduce more-sophisticated model architectures (e.g., ResNet BIBREF32 , Transformer BIBREF18 , etc.) to enhance our model. For now, we focus on the simplest model described above, for the sake of demonstration.
With INLINEFORM0 as the high-level representation of attention, our next step is to reduce it to a weight vector to align information from the context INLINEFORM1 . We apply a max-pooling operation with respect to the context dimension, followed by a linear map to get the logits INLINEFORM2 of the weights DISPLAYFORM0
where INLINEFORM0 is the projection matrix. Then the parsed attention weight INLINEFORM1 is obtained by DISPLAYFORM0
which is used to compute the aligned context embedding
DISPLAYFORM0
Note that here we compute a globally aligned context embedding vector INLINEFORM0 , rather than one for each location in INLINEFORM1 as described in the last section ( INLINEFORM2 ). In the subsequent aggregation operation, INLINEFORM3 is broadcasted to all the locations in INLINEFORM4 . We call this global alignment, to distinguish it from the local alignment strategy described in the last section. Both alignment strategies have their respective merits, and in practice they can be directly combined to produce the final context-aware embedding.
Related Work
Experiments
Experimental setup
We consider three benchmark datasets: ( INLINEFORM0 ) Cora, a paper citation network with text information, built by BIBREF44 . We prune the dataset so that it only has papers on the topic of machine learning. ( INLINEFORM1 ) Hepth, a paper citation network from Arxiv on high energy physics theory, with paper abstracts as text information. ( INLINEFORM2 ) Zhihu, a Q&A network dataset constructed by BIBREF9 , which has 10,000 active users with text descriptions and their collaboration links. Summary statistics of these three datasets are summarized in Table . Pre-processing protocols from prior studies are used for data preparation BIBREF10 , BIBREF34 , BIBREF9 .
For quantitative evaluation, we tested our model on the following tasks: ( INLINEFORM0 ) Link prediction, where we deliberately mask out a portion of the edges to see if the embedding learned from the remaining edges can be used to accurately predict the missing edges. ( INLINEFORM1 ) Multi-label node classification, where we use the learned embedding to predict the labels associated with each node. Note that the label information is not used in our embedding. We also carried out ablation study to identify the gains. In addition to the quantitative results, we also visualized the embedding and the attention matrices to qualitatively verify our hypotheses.
For the link prediction task, we adopt the area under the curve (AUC) score to evaluate the performance, AUC is employed to measure the probability that vertices in existing edges are more similar than those in the nonexistent edge. For each training ratio, the experiment is executed 10 times and the mean AUC scores are reported, where higher AUC indicates better performance. For multi-label classification, we evaluate the performance with Macro-F1 scores. The experiment for each training ratio is also executed 10 times and the average Macro-F1 scores are reported, where a higher value indicates better performance.
To demonstrate the effectiveness of the proposed solutions, we evaluated our model along with the following strong baselines. ( INLINEFORM0 ) Topology only embeddings: MMB BIBREF45 , DeepWalk BIBREF1 , LINE BIBREF33 , Node2vec BIBREF46 . ( INLINEFORM1 ) Joint embedding of topology & text: Naive combination, TADW BIBREF5 , CENE BIBREF6 , CANE BIBREF9 , WANE BIBREF10 , DMTE BIBREF34 . A brief summary of these competing models is provided in the Supplementary Material (SM).
Results
We consider two variants of our model, denoted as GANE-OT and GANE-AP. GANE-OT employs the most basic OT-based attention model, specifically, global word-by-word alignment model; while GANE-AP additionally uses a one-layer convolutional neural network for the attention parsing. Detailed experimental setups are described in the SM.
Tables and summarize the results from the link-prediction experiments on all three datasets, where a different ratio of edges are used for training. Results from models other than GANE are collected from BIBREF9 , BIBREF10 and BIBREF34 . We have also repeated these experiments on our own, and the results are consistent with the ones reported. Note that BIBREF34 did not report results on DMTE. Both GANE variants consistently outperform competing solutions. In the low-training-sample regime our solutions lead by a large margin, and the performance gap closes as the number of training samples increases. This indicates that our OT-based mutual attention framework can yield more informative textual representations than other methods. Note that GANE-AP delivers better results compared with GANE-OT, suggesting the attention parsing mechanism can further improve the low-level mutual attention matrix. More results on Cora and Hepth are provided in the SM.
To further evaluate the effectiveness of our model, we consider multi-label vertex classification. Following the setup described in BIBREF9 , we first computed all context-aware embeddings. Then we averaged over each node's context-aware embeddings with all other connected nodes, to obtain a global embedding for each node, i.e., INLINEFORM0 , where INLINEFORM1 denotes the degree of node INLINEFORM2 . A linear SVM is employed, instead of a sophisticated deep classifier, to predict the label attribute of a node. We randomly sample a portion of labeled vertices with embeddings ( INLINEFORM3 ) to train the classifier, using the rest of the nodes to evaluate prediction accuracy. We compare our results with those from other state-of-the-art models in Table . The GANE models delivered better results compared with their counterparts, lending strong evidence that the OT attention and attention parsing mechanism promise to capture more meaningful representations.
We further explore the effect of INLINEFORM0 -gram length in our model (i.e., the filter size for the covolutional layers used by the attention parsing module). In Figure FIGREF39 we plot the AUC scores for link prediction on the Cora dataset against varying INLINEFORM1 -gram length. The performance peaked around length 20, then starts to drop, indicating a moderate attention span is more preferable. Similar results are observed on other datasets (results not shown). Experimental details on the ablation study can be found in the SM.
Qualitative Analysis
We employed t-SNE BIBREF47 to project the network embeddings for the Cora dataset in a two-dimensional space using GANE-OT, with each node color coded according to its label. As shown in Figure FIGREF40 , papers clustered together belong to the same category, with the clusters well-separated from each other in the network embedding space. Note that our network embeddings are trained without any label information. Together with the label classification results, this implies our model is capable of extracting meaningful information from both context and network topological.
To verify that our OT-based attention mechanism indeed produces sparse attention scores, we visualized the OT attention matrices and compared them with those simarlity-based attention matrices (e.g., WANE). Figure FIGREF44 plots one typical example. Our OT solver returns a sparse attention matrix, while dot-product-based WANE attention is effectively dense. This underscores the effectiveness of OT-based attention in terms of noise suppression.
Conclusion
We have proposed a novel and principled mutual-attention framework based on optimal transport (OT). Compared with existing solutions, the attention mechanisms employed by our GANE model enjoys the following benefits: (i) it is naturally sparse and self-normalized, (ii) it is a global sequence matching scheme, and (iii) it can capture long-term interactions between two sentences. These claims are supported by experimental evidence from link prediction and multi-label vertex classification. Looking forward, our attention mechanism can also be applied to tasks such as relational networks BIBREF15 , natural language inference BIBREF11 , and QA systems BIBREF48 .
Acknowledgments
This research was supported in part by DARPA, DOE, NIH, ONR and NSF. Appendix Competing models Topology only embeddings Mixed Membership Stochastic Blockmodel (MMB) BIBREF45 : a graphical model for relational data, each node randomly select a different "topic" when forming an edge. DeepWalk BIBREF1 : executes truncated random walks on the graph, and by treating nodes as tokens and random walks as natural language sequences, the node embedding are obtained using the SkipGram model BIBREF27 . Node2vec BIBREF46 : a variant of DeepWalk by executing biased random walks to explore the neighborhood (e.g., Breadth-first or Depth-first sampling). Large-scale Information Network Embedding (LINE) BIBREF33 : scalable network embedding scheme via maximizing the joint and conditional likelihoods. Joint embedding of topology & text Naive combination BIBREF9 : direct combination of the structure embedding and text embedding that best predicts edges. Text-Associated DeepWalk (TADW) BIBREF5 : reformulating embedding as a matrix factorization problem, and fused text-embedding into the solution. Content-Enhanced Network Embedding (CENE) BIBREF6 : treats texts as a special kind of nodes. Context-Aware Network Embedding (CANE) BIBREF9 : decompose the embedding into context-free and context-dependent part, use mutual attention to address the context-dependent embedding. Word-Alignment-based Network Embedding (WANE) BIBREF10 : Using fine-grained alignment to improve context-aware embedding. Diffusion Maps for Textual network Embedding (DMTE) BIBREF34 : using truncated diffusion maps to improve the context-free part embedding in CANE. Complete Link prediction results on Cora and Hepth The complete results for Cora and Hepth are listed in Tables and . Results from models other than GANE are collected from BIBREF9 , BIBREF10 , BIBREF34 . We have also repeated these experiments on our own, the results are consistent with the ones reported. Note that DMTE did not report results on Hepth BIBREF34 . Negative sampling approximation In this section we provide a quick justification for the negative sampling approximation. To this end, we first briefly review noise contrastive estimation (NCE) and how it connects to maximal likelihood estimation, then we establish the link to negative sampling. Interested readers are referred to BIBREF50 for a more thorough discussion on this topic. Noise contrastive estimation. NCE seeks to learn the parameters of a likelihood model INLINEFORM0 by optimizing the following discriminative objective: J() = uipd[p(y=1|ui,v) - K Eu'pn [p(y=0|u,v)]], where INLINEFORM1 is the label of whether INLINEFORM2 comes from the data distribution INLINEFORM3 or the tractable noise distribution INLINEFORM4 , and INLINEFORM5 is the context. Using the Monte Carlo estimator for the second term gives us J() = uipd[p(y=1|ui,v) - k=1K [p(y=0|uk,v)]], uk iid pn. Since the goal of INLINEFORM6 is to predict the label of a sample from a mixture distribution with INLINEFORM7 from INLINEFORM8 and INLINEFORM9 from INLINEFORM10 , plugging the model likelihood and noise likelihood into the label likelihood gives us p(y=1;u,v) = p(u|v)p(u|v) + K pn(u|v), p(y=0;u,v) = K pn(u|v)p(u|v) + K pn(u|v). Recall INLINEFORM11 takes the following softmax form DISPLAYFORM0 NCE treats INLINEFORM12 as an learnable parameter and optimized along with INLINEFORM13 . One key observation is that, in practice, one can safely clamp INLINEFORM14 to 1, and the NCE learned model ( INLINEFORM15 ) will self-normalize in the sense that INLINEFORM16 . As such, one can simply plug INLINEFORM17 into the above objective. Another key result is that, as INLINEFORM18 , the gradient of NCE objective recovers the gradient of softmax objective INLINEFORM19 BIBREF49 . Negative sampling as NCE. If we set INLINEFORM20 and let INLINEFORM21 be the uniform distribution on INLINEFORM22 , we have DISPLAYFORM1 where INLINEFORM23 is the sigmoid function. Plugging this back to the INLINEFORM24 covers the negative sampling objective Eqn (6) used in the paper. Combined with the discussion above, we know Eqn (6) provides a valid approximation to the INLINEFORM25 -likelihood in terms of the gradient directions, when INLINEFORM26 is sufficiently large. In this study, we use INLINEFORM27 negative sample for computational efficiency. Using more samples did not significantly improve our results (data not shown). Experiment Setup We use the same codebase from CANE BIBREF9 . The implementation is based on TensorFlow, all experiments are exectuted on a single NVIDIA TITAN X GPU. We set embedding dimension to INLINEFORM28 for all our experiments. To conduct a fair comparison with the baseline models, we follow the experiment setup from BIBREF10 . For all experiments, we set word embedding dimension as 100 trained from scratch. We train the model with Adam optimizer and set learning rate INLINEFORM29 . For GANE-AP model, we use best filte size INLINEFORM30 for convolution from our abalation study. Ablation study setup To test how the INLINEFORM31 -gram length affect our GANE-AP model performance, we re-run our model with different choices of INLINEFORM32 -gram length, namely, the window size in convolutional layer. Each experiment is repeated for 10 times and we report the averaged results to eliminate statistical fluctuations.

Question: Which of their proposed attention methods works better overall?
Answer: attention parsing

Question: Which dataset of texts do they use?
Answer: Cora, Hepth, Zhihu

Question: Do they measure how well they perform on longer sequences specifically?
Answer: Yes

Question: Which other embeddings do they compare against?
Answer: MMB, DeepWalk, LINE,  Node2vec, TADW, CENE, CANE, WANE, DMTE