Abstract: Constituting highly informative network embeddings is an important tool for network analysis. It encodes network topology, along with other useful side information, into low-dimensional node-based feature representations that can be exploited by statistical modeling. This work focuses on learning context-aware network embeddings augmented with text data. We reformulate the network-embedding problem, and present two novel strategies to improve over traditional attention mechanisms: ($i$) a content-aware sparse attention module based on optimal transport, and ($ii$) a high-level attention parsing module. Our approach yields naturally sparse and self-normalized relational inference. It can capture long-term interactions between sequences, thus addressing the challenges faced by existing textual network embedding schemes. Extensive experiments are conducted to demonstrate our model can consistently outperform alternative state-of-the-art methods.

Introduction:  When performing network embedding, one maps network nodes into vector representations that reside in a low-dimensional latent space. Such techniques seek to encode topological information of the network into the embedding, such as affinity BIBREF0 , local interactions (e.g, local neighborhoods) BIBREF1 , and high-level properties such as community structure BIBREF2 . Relative to classical network-representation learning schemes BIBREF3 , network embeddings provide a more fine-grained representation that can be easily repurposed for other downstream applications (e.g., node classification, link prediction, content recommendation and anomaly detection). For real-world networks, one naturally may have access to rich side information about each node. Of particular interest are textual networks, where the side information comes in the form of natural language sequences BIBREF4 . For example, user profiles or their online posts on social networks (e.g., Facebook, Twitter), and documents in citation networks (e.g., Cora, arXiv). The integration of text information promises to significantly improve embeddings derived solely from the noisy, sparse edge representations BIBREF5 . Recent work has started to explore the joint embedding of network nodes and the associated text for abstracting more informative representations. BIBREF5 reformulated DeepWalk embedding as a matrix factorization problem, and fused text-embedding into the solution, while BIBREF6 augmented the network with documents as auxiliary nodes. Apart from direct embedding of the text content, one can first model the topics of the associated text BIBREF7 and then supply the predicted labels to facilitate embedding BIBREF8 . Many important downstream applications of network embeddings are context-dependent, since a static vector representation of the nodes adapts to the changing context less effectively BIBREF9 . For example, the interactions between social network users are context-dependent (e.g., family, work, interests), and contextualized user profiling can promote the specificity of recommendation systems. This motivates context-aware embedding techniques, such as CANE BIBREF9 , where the vector embedding dynamically depends on the context. For textual networks, the associated texts are natural candidates for context. CANE introduced a simple mutual attention weighting mechanism to derive context-aware dynamic embeddings for link prediction. Following the CANE setup, WANE BIBREF10 further improved the contextualized embedding, by considering fine-grained text alignment. Despite the promising results reported thus far, we identify three major limitations of existing context-aware network embedding solutions. First, mutual (or cross) attentions are computed from pairwise similarities between local text embeddings (word/phrase matching), whereas global sequence-level modeling is known to be more favorable across a wide range of NLP tasks BIBREF11 , BIBREF12 , BIBREF13 , BIBREF14 . Second, related to the above point, low-level affinity scores are directly used as mutual attention without considering any high-level parsing. Such an over-simplified operation denies desirable features, such as noise suppression and relational inference BIBREF15 , thereby compromising model performance. Third, mutual attention based on common similarity measures (e.g., cosine similarity) typically yields dense attention matrices, while psychological and computational evidence suggests a sparse attention mechanism functions more effectively BIBREF16 , BIBREF17 . Thus such naive similarity-based approaches can be suboptimal, since they are more likely to incorporate irrelevant word/phrase matching.  This work represents an attempt to improve context-aware textual network embedding, by addressing the above issues. Our contributions include: ( INLINEFORM0 ) We present a principled and more-general formulation of the network embedding problem, under reproducing kernel Hilbert spaces (RKHS) learning; this formulation clarifies aspects of the existing literature and provides a flexible framework for future extensions. ( INLINEFORM1 ) A novel global sequence-level matching scheme is proposed, based on optimal transport, which matches key concepts between text sequences in a sparse attentive manner. ( INLINEFORM2 ) We develop a high-level attention-parsing mechanism that operates on top of low-level attention, which is capable of capturing long-term interactions and allows relational inference for better contextualization. We term our model Global Attention Network Embedding (GANE). To validate the effectiveness of GANE, we benchmarked our models against state-of-the-art counterparts on multiple datasets. Our models consistently outperform competing methods.

Question: What were the sizes of the test sets?
Answer: Test set 1 contained 57 drug labels and 8208 sentences and test set 2 contained 66 drug labels and 4224 sentences