Decentralized Knowledge Graph Representation Learning

1 INTRODUCTION . Knowledge graphs ( KGs ) support many data-driven applications ( Ji et al. , 2020 ) . Recently , learning low-dimensional representations ( a.k.a . embeddings ) of entities and relations in KGs has been increasingly given attentions ( Rossi et al. , 2020 ) . We find that existing models for KG representation learning share similar characteristics to those for word representation learning . For example , TransE ( Bordes et al. , 2013 ) , a well-known translational KG embedding model , interprets a triple ( e1 , r , e2 ) as e1 + r ≈ e2 , where e1 , e2 , r denote subject , object and their relationship , respectively , and the boldfaces denote the corresponding embeddings . If we view e1 as a word in sentences , and e2 as well as many other objects of e1 as the context words , then TransE and many KG embedding models ( Wang et al. , 2014 ; Dettmers et al. , 2018 ; Nguyen et al. , 2018 ; Kazemi & Poole , 2018 ; Sun et al. , 2019 ) , learn representations in a form simaliar to that used in Skip-gram ( Mikolov et al. , 2013a ) , where the input representation is learned to predict the context ( i.e. , neighbors ) representations . Recently , many graph neural networks ( GNNs ) based models for KG representation learning ( Wang et al. , 2018 ; Schlichtkrull et al. , 2018 ; Cao et al. , 2019 ; Wu et al. , 2019 ; Sun et al. , 2020 ; Vashishth et al. , 2020 ) have achieved state-of-the-art performance in KG-related tasks such as entity alignment and entity prediction . Those models learn KG representations in a CBOW ( continuous bag-of-words ) ( Mikolov et al. , 2013a ) manner , in which the context entities are aggregated to predict the target . But they also consider the representation of an entity itself when aggregating the neighborhood information . This nature prevents those models ( e.g. , GCN ( Kipf & Welling , 2017 ) and GAT ( Velickovic et al. , 2018 ) ) to be generalized to represent unseen entities . In many cases , the entities in prevalent KG-related tasks do not have self features . This motivates us to learn entity representations from and only from their context neighbors . We propose a decentralized KG representation learning approach , decentRL . The key idea of decentRL is to decentralize the semantic information of entities over only their neighbors ( i.e. , distributed context vector in CBOW ( Mikolov et al. , 2013b ) ) , which can be easily implemented by representing each entity through averaging its neighbor embeddings . In this paper , we look for a more efficient but still simple way to realize this concept on the most popular graph attention network ( GAT ) ( Velickovic et al. , 2018 ) , as well as its many variants ( Sun et al. , 2020 ; Vashishth et al. , 2020 ) . We illustrate the methodology by decentralized attention network ( DAN ) , which is based on the vallina GAT . DAN is able to support KG representation learning for unseen entities with only structure information , which is essentially different from the way of using self features ( e.g. , attribute information ) in the existing graph embedding models ( Hamilton et al. , 2017 ; Bojchevski & Günnemann , 2018 ; Hettige et al. , 2020 ) . Furthermore , the neighbors in DAN serve as an integrity to give attentions , which means that DAN is more robust and more expressive compared with conventional graph attention mechanism ( Velickovic et al. , 2018 ) . Another key problem in decentralized KG representation learning is how to estimate and optimize the output embeddings . If we distribute the information of an entity over its neighbors , the original embedding of this entity ei also learns how to effectively participate in the aggregations of its different neighbors conversely . Suppose that we have obtained an output representation gi from DAN for entity ei , we can simply estimate and optimize gi by aligning it with ei . But directly minimizing the L1/L2 distance between gi and ei may be insufficient . Specifically , these two embeddings have completely different roles and functions in the model , and the shared information may not reside in the same dimensions . Therefore , maximizing the mutual information between them is a better choice . Different from the existing works like MINE ( Belghazi et al. , 2018 ) or InfoNCE ( van den Oord et al. , 2018 ) , in this paper , we design a self knowledge distillation algorithm , called auto-distiller . It alternately optimizes gi and its potential target ei , such that gi can automatically and continuously distill knowledge from the original representation ei across different batches . The main contributions of this paper are listed as follows . ( 1 ) We propose decentralized KG representation learning , and present DAN as the prototype of graph attention mechanism under the open-world setting . ( 2 ) We design an efficient knowledge distillation algorithm to support DAN for generating representations of unseen entities . ( 3 ) We implement an end-to-end framework based on DAN and auto-distiller . The experiments show that it achieved superior performance on two prevalent KG representation learning tasks ( i.e. , entity alignment and entity prediction ) , and also significantly outperformed those cutting-edge models under the open-world setting . 2 BACKGROUND . Knowledge Graph . A KG can be viewed as a multi-relational graph , in which nodes represent entities in the real world and edges have specific labels to represent different relationships between entities . Formally , we define a KG as a 3-tuple G = ( E , R , T ) , with E and R denoting the sets of entities and relationships , respectively . T is the set of relational triples . KG Representation Learning . Conventional models are mainly based on the idea of Skip-gram . According to the types of their score functions , these models can be divided into three categories : translational models ( e.g. , TransE ( Bordes et al. , 2013 ) and TransR ( Lin et al. , 2015a ) ) , semantic matching models ( e.g. , DistMult ( Yang et al. , 2015 ) and ComplEx ( Trouillon et al. , 2016 ) ) and neural models ( e.g. , ConvE ( Dettmers et al. , 2018 ) and RSN ( Guo et al. , 2019 ) ) . We refer interested readers to the surveys ( Wang et al. , 2017 ; Ji et al. , 2020 ) for details . Recently , GNN-based models receive great attentions in this field , which are closely related to this paper . Specifically , R-GCN ( Schlichtkrull et al. , 2018 ) , AVR-GCN ( Ye et al. , 2019 ) and CompGCN ( Vashishth et al. , 2020 ) introduce different relation-specific composition operations to combine neighbors and the corresponding relations before neighbor aggregation . RDGCN ( Wu et al. , 2019 ) refactors KGs as dual relation graphs ( Monti et al. , 2018 ) where edge labels are represented as nodes for graph convolution . All the aforementioned GNN-based models choose GCNs and/or GATs to aggregate the neighbors of an entity , in which an identity matrix is added to the adjacency matrix . This operation is helpful when elements have self features , but poses a problem in learning the representations of unseen entities where no self features are attached to them . Differently , decentRL fully relies on the neighbor context to attend to the neighbors of each entity in linear complexity , which is efficient and easy to be deployed . Entity Alignment . Entity alignment aims to find the potentially aligned entity pairs in two different KGs G1 = ( E1 , R1 , T1 ) and G2 = ( E2 , R2 , T2 ) , given a limited number of aligned pairs as training data S ⊂ E1 × E2 . Oftentimes , G1 , G2 are merged to a joint KG G = ( E , R , T ) , which enables the models learn representations in a unified space . Entity Prediction . Entity prediction ( a.k.a . KG completion ( Bordes et al. , 2013 ) ) seeks to find the missing subject e1 or object e2 , given an incomplete relation triple ( ? , r , e2 ) or ( e1 , r , ? ) . It is worth noting that the performance on the entity prediction task may be greatly improved by complex deep networks , as it relies on the predictive ability rather than the embedding quality ( Guo et al. , 2019 ) . Hence , many cutting-edge models can not obtain promising results in entity alignment ( Guo et al. , 2019 ; Sun et al. , 2020 ) . Differently , entity alignment directly compares the distance of learned entity embeddings , which clearly reflects the quality of output representations . Few models demonstrate consistently good performance on both tasks , whereas decentRL is capable of achieving competitive , even better , performance compared with respective state-of-the-art models . 3 DECENTRALIZED REPRESENTATION LEARNING . In the decentralized setting , the representation of an entity ei is aggregated from and only from its neighbors Ni = { e1 , e2 , . . . , e|Ni| } . As it may have many neighbors that are unequally informative ( Velickovic et al. , 2018 ) , involving attention mechanism is a good choice . 3.1 GRAPH ATTENTION NETWORKS . We start by introducing the Graph attention network ( GAT ) ( Velickovic et al. , 2018 ) , which leverages linear self attention to operate spatially close neighbors . For an entity ei , GAT aggregates the representations of its neighbors Ni and itself into a single representation ci as follows : ci = ∑ ej∈Ni∪ { ei } aijWej , ( 1 ) where aij is the learnable attention score from ei to ej , and W is the weight matrix . To obtain aij , a linear attention mechanism is used here : aij = softmax ( σ ( aT [ W1ei ‖W2ej ] ) ) , ( 2 ) where a is a weight vector to convert the concatenation of two embeddings into a scalar attention score , and ‖ denotes the concatenation operation . W1 and W2 are two weight matrices . σ is the activation function , usually being LeakyReLU ( Xu et al. , 2015 ) . GAT computes the attention score of an entity ei to its neighbors in linear complexity , which is very efficient when being applied to large-scale graphs . 3.2 DECENTRALIZED ATTENTION NETWORKS . Intuitively , if ei is the embedding of an unseen entity , it is rarely useful in computing the attention scores ( as it is just a randomly initialized vector ) . Thus , purely relying on its neighbors may be a good choice . Specifically , to obtain the decentralized attention scores , one may simply sum all the attention scores from other neighbors a′ij = softmax ( ∑ ek∈Ni\ { ej } akj ) . However , it would lead to a problem that this sum only represents the attention of each neighbor to ej . In this case , a high attention score from one neighbor ek to ej can dominate the value of a′ij , but it does not mean that ej is more important for ei . Therefore , all neighbors should act as an integrity in giving attentions . Towards this end , we propose decentralized attention networks ( DANs ) . Formally , to obtain the decentralized attention weight aij , we have to feed the attention layer with two types of input : the neighbor context vector ni ( i.e. , query ) , and the candidate neighbor embedding ej ( i.e. , key and value ) . Separately controlling the iterations of these two variables in a multi-layer model is evidently inefficient . Instead , we realize this operation by involving a second-order attention mechanism . For layer k , DAN calculates the decentralized attention score akij as follows : akij = softmax ( σ ( aTk [ W k 1d k−1 i ‖W k 2d k−2 j ] ) ) , ( 3 ) where dk−1i , d k−2 j denote the output embeddings of layer k − 1 for ei and of layer k − 2 for ej , respectively . If we regard dk−1i as the neighbor aggregation of layer k−1 for ei , then d k−2 j is exactly the embedding of ej used in summing dk−1i . In this case , a k ij can represent the attention weight of ei ’ s neighbor context to ej . Then , we can obtain the output of layer k by : dki = ∑ ej∈Ni akijW kdk−2j . ( 4 ) It is worth noting that we perform convolutions on layer k − 2 , as the score akij is attended to the neighbor representations in layer k − 2 . This keeps the consistency and ensures the output representations are consecutive . Also , it enhances the correlation of output in different layers , and forms the second-order graph attention mechanism . For the first layer of DAN , we initialize d0i and d −1 j as follows : d0i = 1 |Ni| ∑ ej∈Ni W0ej , d −1 j = ej . ( 5 ) Here , we simply use a mean aggregator to obtain the decentralized embedding d0i of layer 0 , but other aggregators like pooling may be employed as well . This simple mean aggregator can also be regarded as a CBOW model with dynamic window size . For the architecture and implementation of DAN , please refer to Appendix A .

Summary: This paper presents a method for knowledge graph embedding based on graph attention networks (GAT). The key idea is to avoid using the information for a node (i.e., its representation vectors) when computing the attention weights for the neighbors of the node. The paper argues that this approach can better generalize to unseen nodes where no pre-defined features/information is available. As such, the paper does not include the representations for a node $e$ from prior layers in the aggregations to compute $e$'s representations in the next layers, leveraging the representation vectors of the nodes from prior layers to obtain attention weights for the current layer. The paper also proposes to a self-learning method to learn the parameters by optimizing the mutual information of the final and initial embedding vectors for the nodes. A distillation approach is also employed to use the initial embedding vectors as the teachers and the final embedding vectors as the students. The proposed method is applied to two downstream tasks, i.e., entity alignment and entity prediction, leading to competitive performance with many prior works (the learned node embeddings still need to be aligned using task-specific losses). Some experiments on unseen entities and ablation studies are also conducted to demonstrate the benefits of the proposed method.