SENSE: SEMANTICALLY ENHANCED NODE SEQUENCE EMBEDDING

1 INTRODUCTION . Accurately learning vector embeddings for a sequence of nodes in a graph is critical to many scenarios , e.g. , a set of Web pages regarding one specific topic that are linked together . Such a task is challenging as : ( i ) the embeddings may have to capture graph structure along with any available textual descriptions of the nodes , and moreover , ( ii ) nodes of interest may be associated with a specific order . For instance , ( i ) for a set of Wikipedia pages w.r.t . a topic , there exists a recommended reading sequence ; ( ii ) an application may consist of a set of services/functions , which must be executed in a particular order ( workflow composability ) ; ( iii ) in source routing , the sender of a packet on the Internet specifies the path that the packet takes through the network or ( iv ) the general representation of any path in a graph or a network , e.g. , shortest path . Node sequence embedding , thus , requires us to ( i ) learn embeddings for each individual node of the graph and ( ii ) compose them together to represent their sequences . To learn the right representation of individual nodes and also their sequences , we need to understand how these nodes are correlated with each other both functionally and structurally . A lot of work has only gone into learning single node embeddings ( i.e. , where node sequence length is 1 ) , as they are essential in feature representations for applications like multi-label classification or link prediction . For instance , algorithms in Perozzi et al . ( 2014 ) , Grover & Leskovec ( 2016 ) , Tang et al . ( 2015 ) and others try to extract features purely from the underlying graph structure ; algorithms in Le & Mikolov ( 2014 ) , Mikolov et al . ( 2013 ) and others learn vector representations of documents sharing a common vocabulary set . However , many applications would potentially benefit from representations that are able to capture both textual descriptions and the underlying graph structure simultaneously . For example , ( 1 ) classification of nodes in a network not only depends on their inter-connections ( i.e. , graph structure ) , but also nodes ’ intrinsic properties ( i.e. , their textual descriptions ) ; ( 2 ) for product recommendations , if the product is new , it may not have many edges since not many users have interacted with it ; however , using the textual descriptions along with the graph structure allows for efficient bootstrapping of the recommendation service . For general case of sequence lengths greater than 1 , despite the importance in applications like workflow composability described above , there is generally a lack of efficient solutions . Intuitively , we can concatenate or add all involved node vectors ; however , such a mechanism either takes too much space or loses the sequence information ; thus unable to represent node sequences properly . We aim to learn node sequence embeddings by first first addressing the single node embedding issue , as a special case of node sequence embedding , by considering both the textual descriptions and the graph structure . We seek to answer two questions : How should we combine these two objectives ? What framework should we use for feature learning ? Works that jointly address these two questions either investigate them under different problem settings ( Cao et al. , 2017 ; Xiao et al. , 2017 ) , under restricted learning models ( Le & Lauw , 2014 ) , ignore the word context within the document ( Liao et al. , 2017 ; Wang et al. , 2017 ) , do not co-learn text and graph patterns ( Li et al. , 2016 ) or only consider linear combinations of text and graph ( Garcia Duran & Niepert , 2017 ) ; this is elaborated further in Section 2 . In contrast , we propose a generic neural-network-based model called SENSES ( Semantically Enhanced Node Sequence Embeddings - for Single nodes ) for computing vector representations of nodes with additional semantic information in a graph . SENSE-S is built on the foundation of skip-gram models . However , SENSE-S is significantly different from classic skipgram models in the following aspects : ( i ) For each word φ in the textual description of node v in the given graph , neighboring words of φ within v ’ s textual description and neighboring nodes of v within the graph are sampled at the same time . ( ii ) The text and graph inputs are both reflected in the output layer in the form of probabilities of co-occurrence ( in graph or text ) . ( iii ) Moreover , this joint optimization problem offers an opportunity to leverage the synergy between the graph and text inputs to ensure faster convergence . We evaluate the generated vectors on ( i ) Wikispeedia ( 2009 ) to show that our SENSE-S model improves multi-label classification accuracy by up to 50 % and ( ii ) Physics Citation dataset ( Leskovec & Krevl , 2014 ) to show that SENSE-S improves link prediction accuracy by up to 78 % over the state-of-the-art . Next , we propose SENSE for general feature representation of a sequence of nodes . This problem is more challenging in that ( i ) besides the original objectives in SENSE-S , we now face another representation goal , i.e. , sequence representation while preserving the node order ; ( ii ) it is important to represent the sequence in a compact manner ; and ( iii ) more importantly , given a sequence vector , we need to be able to decipher which functional nodes are involved and in what order . To this end , we develop efficient schemes to combine individual vectors into complex sequence vectors that address all of the above challenges . The key technique we use here is vector cyclic shifting , and we prove that the different shifted vectors are orthogonal with high probability . This sequence embedding method is also evaluated on the Wikispeedia and Physics Citation datasets , and the accuracy of decoding a node sequence is shown to be close to 100 % when the vector dimension is large . 2 RELATED WORK . We overview the most related works by categorizing them as follows : Learning vector representation from text : Vector representation of words ( Schütze , 1993 ) has been a long standing research topic . It has received significant attention in the recent times due to the advances in deep neural networks ( Bengio et al. , 2003 ; Mikolov et al. , 2009 ; 2013 ) . In particular , these neural-network-based schemes outperform n-gram-based techniques ( Katz , 1987 ; Jelinek & Mercer , 1980 ) significantly as they are able to learn the similarities between words . Furthermore , paragraph2vec ( Le & Mikolov , 2014 ) extends the well-established word2vec ( Mikolov et al. , 2013 ) to learn representations of chunks of text . Learning vector representation from graphs : Lot of research has gone into learning graph representations by translating the network/graph into a set of words or documents ( Perozzi et al. , 2014 ; Pimentel et al. , 2017 ; Ribeiro et al. , 2017 ; Wang et al. , 2016 ; Liu et al. , 2016 ; Tang et al. , 2015 ; Grover & Leskovec , 2016 ) . Generic models incorporating both edge weight and direction information for graph embeddings are proposed in Zhou et al . ( 2017 ) , Tang et al . ( 2015 ) and Grover & Leskovec ( 2016 ) . Specifically , node2vec ( Grover & Leskovec , 2016 ) advances the state-of-the-art in this area by designing flexible node sampling methodologies to allow feature vectors to exhibit different properties . Subgraph2vec ( Narayanan et al. , 2016 ) extends these schemes to learn vector representations of subgraphs . Simpkin et al . ( 2018 ) proposes techniques to represent graph sequences under the assumption that each node is represented by a random binary vector . Learning graph representation with auxiliary information : Broadly speaking , our work falls into the category of node embedding in graphs with auxiliary information . Guo et al . ( 2017 ) , Xie et al . ( 2016 ) , Huang et al . ( 2017a ) and others address the case where nodes are associated with labels . Niepert et al . ( 2016 ) studies graph embedding when node/edge attributes are continuous . Cao et al . ( 2017 ) investigates phrase ambiguity resolution via leveraging hyperlinks . However , all these works operate under information or network constraints . On the other hand , Xiao et al . ( 2017 ) , Yao et al . ( 2017 ) and Wang & Li ( 2016 ) explore embedding strategies in the context of knowledge graphs , where the main goal is to maintain the entity relationships specified by semantic edges . In contrast , we consider a simpler network setting where only nodes are associated with semantic information . EP ( Garcia Duran & Niepert , 2017 ) and GraphSAGE ( Hamilton et al. , 2017 ) learn embeddings for structured graph data . However , the textual similarities are only captured by linear combinations . Planetoid ( Yang et al. , 2016 ) computes node embeddings under semi-supervised settings ; metapath2vec ( Dong et al. , 2017 ) learns embeddings for heterogeneous networks ( node can be author or paper ) ; and Graph-Neural-Network-based embeddings are explored in Kipf & Welling ( 2017 ) and Li et al . ( 2016 ) . However , these papers do not explicitly learn graph structure and text patterns simultaneously , and thus are complementary to SENSE . SNE ( Liao et al. , 2017 ) , SNEA ( Wang et al. , 2017 ) , TADW ( Yang et al. , 2015 ) , HSCA ( Zhang et al. , 2016 ) , AANE ( Huang et al. , 2017b ) , ANRL ( Zhang et al. , 2018 ) and PLANE ( Le & Lauw , 2014 ) are more related to our work . However , unlike SENSE , these do not consider the relative context of the words w.r.t . the document . Furthermore , the objective of PLANE is to maximize the likelihood that neighboring nodes have similar embeddings , which is not always the case in practice because neighboring nodes may be semantically different ; more critically , it relies on strong assumptions of statistical distributions of words and edges in the network . In this regard , we propose a generic embedding scheme that jointly considers network topology as well as the nodes ’ semantic information . 3 SENSE-S : SENSE FOR SINGLE NODE EMBEDDINGS . To embed a general node sequence , we first consider a special case where each node sequence contains only one node . Such single node embedding is referred to as SENSE-S , which jointly learns node representations along with textual descriptions in graphs . 3.1 SENSE-S OBJECTIVE . Let G = ( V , E ) denote a given directed or undirected graph , where V is the set of nodes and E the set of edges . Each node in V is associated with a text description . We aim to embed each node v in V into a feature vector that captures both graphical ( neighboring node inter-connections ) and textual ( semantic meaning of v ) properties . Specifically , let φ denote a word in the text description of node v. Suppose we obtain a set of neighboring nodes of v in graph G via a specific node sampling strategy , e.g. , biased random walk ( Grover & Leskovec , 2016 ) , and a set of neighboring words of φ in the text description of v by a sliding window over consecutive words ( Mikolov et al. , 2013 ) . We then defineNG ( v ) as a probabilistic event of observing the set of neighboring nodes of v in G ( under the chosen model ) and NT ( φ|v ) as an event of observing the set of neighboring words of φ in the text description of v. Let f : v → Rd ( v ∈ V ) be the embedding function that maps each node v in V into a d-dimensional vector . Our goal is to find function f that maximizes : max f ∑ v∈V ∑ φ in v log Pr [ NG ( v ) NT ( φ|v ) |φ , f ( v ) ] . ( 1 ) Since events NG ( v ) and NT ( φ|v ) are independent , ( 1 ) can be rewritten as : max f ∑ v∈V wv log Pr [ NG ( v ) |f ( v ) ] + ∑ v∈V ∑ φ in v log Pr [ NT ( φ|v ) |φ , f ( v ) ] , ( 2 ) where wv is the number of words in the description of v. Given a word in a node , ( 2 ) jointly captures the node neighborhood in the graph and the word neighborhood in text .

Summary: This paper presents two approaches: one called SENSE-S for embedding nodes in attributed networks; the other one called SENSE for embedding a sequence of nodes. SENSE-S follows the structure of Skip-gram model. The main difference is that SENSE-S considers both node and words in node content as input and output for learning their embedding. For generating embedding vector for a sequence of nodes, SENSE takes summation of cyclically shifted unit-vectors constructed by SENSE-S on nodes in a sequence.