Introduction
Reasoning over multi-relational data is a key concept in Artificial Intelligence and knowledge graphs have appeared at the forefront as an effective tool to model such multi-relational data. Knowledge graphs have found increasing importance due to its wider range of important applications such as information retrieval BIBREF0 , natural language processing BIBREF1 , recommender systems BIBREF2 , question-answering BIBREF3 and many more. This has led to the increased efforts in constructing numerous large-scale Knowledge Bases (e.g. Freebase BIBREF4 , DBpedia BIBREF5 , Google's Knowledge graph BIBREF6 , Yago BIBREF7 and NELL BIBREF8 ), that can cater to these applications, by representing information available on the web in relational format.
All knowledge graphs share common drawback of incompleteness and sparsity and hence most existing relational learning techniques focus on using observed triplets in an incomplete graph to infer unobserved triplets for that graph BIBREF9 . Neural embedding techniques that learn vector space representations of entities and relationships have achieved remarkable success in this task. However, these techniques only focus on learning from a single graph. In addition to incompleteness property, these knowledge graphs also share a set of overlapping entities and relationships with varying information about them. This makes a compelling case to design a technique that can learn over multiple graphs and eventually aid in constructing a unified giant graph out of them. While research on learning representations over single graph has progressed rapidly in recent years BIBREF10 , BIBREF6 , BIBREF11 , BIBREF12 , BIBREF13 , BIBREF14 , there is a conspicuous lack of principled approach to tackle the unique challenges involved in learning across multiple graphs.
One approach to multi-graph representation learning could be to first solve graph alignment problem to merge the graphs and then use existing relational learning methods on merged graph. Unfortunately, graph alignment is an important but still unsolved problem and there exist several techniques addressing its challenges BIBREF15 , BIBREF16 , BIBREF17 , BIBREF18 in limited settings. The key challenges for the graph alignment problem emanate from the fact that the real world data are noisy and intricate in nature. The noisy or sparse data make it difficult to learn robust alignment features, and data abundance leads to computational challenges due to the combinatorial permutations needed for alignment. These challenges are compounded in multi-relational settings due to heterogeneous nodes and edges in such graphs.
Recently, deep learning has shown significant impact in learning useful information over noisy, large-scale and heterogeneous graph data BIBREF19 . We, therefore, posit that combining graph alignment task with deep representation learning across multi-relational graphs has potential to induce a synergistic effect on both tasks. Specifically, we identify that a key component of graph alignment process—entity linkage—also plays a vital role in learning across graphs. For instance, the embeddings learned over two knowledge graphs for an actor should be closer to one another compared to the embeddings of all the other entities. Similarly, the entities that are already aligned together across the two graphs should produce better embeddings due to the shared context and data. To model this phenomenon, we propose LinkNBed, a novel deep learning framework that jointly performs representation learning and graph linkage task. To achieve this, we identify key challenges involved in the learning process and make the following contributions to address them:
Knowledge Graph Representation
A knowledge graph $\mathcal {G}$ comprises of set of facts represented as triplets ( $e^s,r,e^o$ ) denoting the relationship $r$ between subject entity $e^s$ and object entity $e^o$ . Associated to this knowledge graph, we have a set of attributes that describe observed characteristics of an entity. Attributes are represented as set of key-value pairs for each entity and an attribute can have null (missing) value for an entity. We follow Open World Assumption - triplets not observed in knowledge graph are considered to be missing but not false. We assume that there are no duplicate triplets or self-loops.
Multi-Graph Relational Learning
Definition. Given a collection of knowledge graphs $\mathcal {G}$ , Multi-Graph Relational Learning refers to the the task of learning information rich representations of entities and relationships across graphs. The learned embeddings can further be used to infer new knowledge in the form of link prediction or learn new labels in the form of entity linkage. We motivate our work with the setting of two knowledge graphs where given two graphs $G_1, G_2 \in \mathcal {G}$ , the task is to match an entity $e_{G_1} \in G_1$ to an entity $e_{G_2} \in G_2$ if they represent the same real-world entity. We discuss a straightforward extension of this
setting to more than two graphs in Section 7. Notations. Let $X$ and $Y$ represent realization of two such knowledge graphs extracted from two different sources. Let $n_e^X$ and $n_e^Y$ represent number of entities in $X$ and $Y$ respectively. Similarly, $n_r^X$ and $n_r^Y$ represent number of relations in $X$ and $Y$ . We combine triplets from both $Y$0 and $Y$1 to obtain set of all observed triplets $Y$2 where $Y$3 is total number of available records across from both graphs. Let $Y$4 and $Y$5 be the set of all entities and all relations in $Y$6 respectively. Let $Y$7 and $Y$8 . In addition to $Y$9 , we also have set of linkage labels $n_e^X$0 for entities between $n_e^X$1 and $n_e^X$2 . Each record in $n_e^X$3 is represented as triplet ( $n_e^X$4 , $n_e^X$5 , $n_e^X$6 ) where $n_e^X$7 when the entities are matched and $n_e^X$8 otherwise.
Proposed Method: LinkNBed 
We present a novel inductive multi-graph relational learning framework that learns a set of aggregator functions capable of ingesting various contextual information for both entities and relationships in multi-relational graph. These functions encode the ingested structural and semantic information into low-dimensional entity and relation embeddings. Further, we use these representations to learn a relational score function that computes how two entities are likely to be connected in a particular relationship. The key idea behind this formulation is that when a triplet is observed, the relationship between the two entities can be explained using various contextual information such as local neighborhood features of both entities, attribute features of both entities and type information of the entities which participate in that relationship.
We outline two key insights for establishing the relationships between embeddings of the entities over multiple graphs in our framework:
Insight 1 (Embedding Similarity): If the two entities $e^X \in X$ and $e^Y \in Y$ represent the same real-world entity then their embeddings $\mathbf {e^X}$ and $\mathbf {e^Y}$ will be close to each other.
Insight 2 (Semantic Replacement): For a given triplet $t = (e^s, r, e^o) \in X$ , denote $g(t)$ as the function that computes a relational score for $t$ using entity and relation embeddings. If there exists a matching entity $e^{s^{\prime }} \in Y$ for $e^s \in X$ , denote $t^{\prime } = (e^{s^{\prime }},r,e^o)$ obtained after replacing $e^s$ with $e^{s^{\prime }}$ . In this case, $g(t) \sim g(t^{\prime })$ i.e. score of triplets $t$ and $g(t)$0 will be similar.
For a triplet $(e^s, r , e^o) \in \mathcal {D}$ , we describe encoding mechanism of LinkNBed as three-layered architecture that computes the final output representations of $\mathbf {z}^{r}, \mathbf {z}^{e^s}, \mathbf {z}^{e^o}$ for the given triplet. Figure 1 provides an overview of LinkNBed architecture and we describe the three steps below:
Atomic Layer
Entities, Relations, Types and Attributes are first encoded in its basic vector representations. We use these basic representations to derive more complex contextual embeddings further.
Entities, Relations and Types. The embedding vectors corresponding to these three components are learned as follows: ves = f(WE es ) veo = f(WE eo )
vr = f(WR r ) vt = f(WT t ) where $\mathbf {v^{e^s}}$ , $\mathbf {v^{e^o}} \in \mathbb {R}^{d}$ . $\mathbf {e^s}$ , $\mathbf {e^o} \in \mathbb {R}^{n}$ are “one-hot" representations of $e^s$ and $e^o$ respectively. $\mathbf {v^{r}} \in \mathbb {R}^{k}$ and $\mathbf {r} \in \mathbb {R}^{m}$ is “one-hot" representation of $r$ . $\mathbf {v^{t}} \in \mathbb {R}^{q}$ and $\mathbf {v^{e^o}} \in \mathbb {R}^{d}$0 is "one-hot" representation of $\mathbf {v^{e^o}} \in \mathbb {R}^{d}$1 . $\mathbf {v^{e^o}} \in \mathbb {R}^{d}$2 , $\mathbf {v^{e^o}} \in \mathbb {R}^{d}$3 and $\mathbf {v^{e^o}} \in \mathbb {R}^{d}$4 are the entity, relation and type embedding matrices respectively. $\mathbf {v^{e^o}} \in \mathbb {R}^{d}$5 is a nonlinear activation function (Relu in our case). $\mathbf {v^{e^o}} \in \mathbb {R}^{d}$6 , $\mathbf {v^{e^o}} \in \mathbb {R}^{d}$7 and $\mathbf {v^{e^o}} \in \mathbb {R}^{d}$8 can be initialized randomly or using pre-trained word embeddings or vector compositions based on name phrases of components BIBREF20 .
Attributes. For a given attribute $a$ represented as key-value pair, we use paragraph2vec BIBREF21 type of embedding network to learn attribute embedding. Specifically, we represent attribute embedding vector as: a = f(Wkey akey + Wval aval )
where $\mathbf {a} \in \mathbb {R}^{y}$ , $\mathbf {a_{key}} \in \mathbb {R}^{u}$ and $\mathbf {a_{val}} \in \mathbb {R}^{v}$ . $\mathbf {W^{key}} \in \mathbb {R}^{y \times u}$ and $\mathbf {W^{val}} \in \mathbb {R}^{y \times v}$ . $\mathbf {a_{key}}$ will be “one-hot" vector and $\mathbf {a_{val}}$ will be feature vector. Note that the dimensions of the embedding vectors do not necessarily need to be the same.
Contextual Layer
While the entity and relationship embeddings described above help to capture very generic latent features, embeddings can be further enriched to capture structural information, attribute information and type information to better explain the existence of a fact. Such information can be modeled as context of nodes and edges in the graph. To this end, we design the following canonical aggregator function that learns various contextual information by aggregating over relevant embedding vectors: c(z) = AGG({z', z' C(z)}) where $\mathbf {c}(z)$ is the vector representation of the aggregated contextual information for component $z$ . Here, component $z$ can be either an entity or a relation. $C(z)$ is the set of components in the context of $z$ and $\mathbf {z^{\prime }}$ correspond to the vector embeddings of those components. AGG is the aggregator function which can take many forms such Mean, Max, Pooling or more complex LSTM based aggregators. It is plausible that different components in a context may have varied impact on the component for which the embedding is being learned. To account for this, we employ a soft attention mechanism where we learn attention coefficients to weight components based on their impact before aggregating them. We modify Eq. "Implementation Details" as:
c(z) = AGG(q(z) * {z', z' C(z)}) where q(z) = (z)z' C(z) (z') and $\theta _z$ 's are the parameters of attention model.
Following contextual information is modeled in our framework:
Entity Neighborhood Context $\mathbf {N_c}(e) \in \mathbb {R}^d$ . Given a triplet $(e^s,r,e^o)$ , the neighborhood context for an entity $e^s$ will be the nodes located near $e^s$ other than the node $e^o$ . This will capture the effect of local neighborhood in the graph surrounding $e^s$ that drives $e^s$ to participate in fact $(e^s,r,e^o)$ . We use Mean as aggregator function. As there can be large number of neighbors, we collect the neighborhood set for each entity as a pre-processing step using a random walk method. Specifically, given a node $e$ , we run $k$ rounds of random-walks of length $(e^s,r,e^o)$0 following BIBREF22 and create set $(e^s,r,e^o)$1 by adding all unique nodes visited across these walks. This context can be similarly computed for object entity.
Entity Attribute Context $\mathbf {A_c}(e) \in \mathbb {R}^y$ . For an entity $e$ , we collect all attribute embeddings for $e$ obtained from Atomic Layer and learn aggregated information over them using Max operator given in Eq. "Implementation Details" .
Relation Type Context $\mathbf {T_c}(r) \in \mathbb {R}^q$ . We use type context for relation embedding i.e. for a given relationship $r$ , this context aims at capturing the effect of type of entities that have participated in this relationship. For a given triplet $(e^s, r , e^o)$ , type context for relationship $r$ is computed by aggregation with mean over type embeddings corresponding to the context of $r$ . Appendix C provides specific forms of contextual information.
Representation Layer
Having computed the atomic and contextual embeddings for a triplet $(e^s, r, e^o)$ , we obtain the final embedded representations of entities and relation in the triplet using the following formulation: $ \mathbf {z^{e^s}} &= \sigma (\underbrace{\mathbf {W_1v^{e^s}}}_\text{Subject Entity Embedding} + \underbrace{\mathbf {W_2 N_c}(e^s)}_\text{Neighborhood Context}\\ &+ \underbrace{\mathbf {W_3 A_c}(e^s))}_\text{Subject Entity Attributes} $ $ \mathbf {z^{e^o}} &= \sigma (\underbrace{\mathbf {W_1v^{e^o}}}_\text{Object Entity Embedding} + \underbrace{\mathbf {W_2 N_c}(e^o)}_\text{Neighborhood Context}\\ &+ \underbrace{\mathbf {W_3 A_c}(e^o))}_\text{Object Entity Attributes} $
zr = (W4vrRelation Embedding + W5 Tc(r))Entity Type Context
where $\mathbf {W_1}, \mathbf {W_2} \in \mathbb {R}^{d \times d}$ , $\mathbf {W_3} \in \mathbb {R}^{d \times y}$ , $\mathbf {W_4} \in \mathbb {R}^{d \times k}$ and $\mathbf {W_5} \in \mathbb {R}^{d \times q}$ . $\sigma $ is nonlinear activation function – generally Tanh or Relu.
Following is the rationale for our formulation: An entity's representation can be enriched by encoding information about the local neighborhood features and attribute information associated with the entity in addition to its own latent features. Parameters $\mathbf {W_1}, \mathbf {W_2}, \mathbf {W_3}$ learn to capture these different aspects and map them into the entity embedding space. Similarly, a relation's representation can be enriched by encoding information about entity types that participate in that relationship in addition to its own latent features. Parameters $\mathbf {W_4}, \mathbf {W_5}$ learn to capture these aspects and map them into the relation embedding space. Further, as the ultimate goal is to jointly learn over multiple graphs, shared parameterization in our model facilitate the propagation of information across graphs thereby making it a graph-independent inductive model. The flexibility of the model stems from the ability to shrink it (to a very simple model considering atomic entity and relation embeddings only) or expand it (to a complex model by adding different contextual information) without affecting any other step in the learning procedure.
Relational Score Function
Having observed a triplet $(e^s,r, e^o)$ , we first use Eq. 7, 8 and 9 to compute entity and relation representations. We then use these embeddings to capture relational interaction between two entities using the following score function $g(\cdot )$ : g(es, r, eo) = (zrT (zes zeo)) where $\mathbf {z}^{r}, \mathbf {z}^{e^s}, \mathbf {z}^{e^o} \in \mathbb {R}^d$ are $d$ -dimensional representations of entity and relationships as described below. $\sigma $ is the nonlinear activation function and $\odot $ represent element-wise product.
Objective Function
The complete parameter space of the model can be given by: $\mathbf {\Omega = \lbrace \lbrace W_i\rbrace _{i=1}^5, W^E, W^R, W^{key}, W^{val}, W^t ,\Theta \rbrace }$ . To learn these parameters, we design a novel multi-task objective function that jointly trains over two graphs. As identified earlier, the goal of our model is to leverage the available linkage information across graphs for optimizing the entity and relation embeddings such that they can explain the observed triplets across the graphs. Further, we want to leverage these optimized embeddings to match entities across graphs and expand the available linkage information. To achieve this goal, we define following two different loss functions catering to each learning task and jointly optimize over them as a multi-task objective to learn model parameters:
Relational Learning Loss. This is conventional loss function used to learn knowledge graph embeddings. Specifically, given a p-th triplet $(e^s, r, e^o)_p$ from training set $\mathcal {D}$ , we sample $C$ negative samples by replacing either head or tail entity and define a contrastive max margin function as shown in BIBREF20 :
$$\begin{split} L_{rel} &= \sum \limits _{c=1}^{C} \max (0, \gamma - g(e^s_p,r_p,e^o_p) \\ &+ g^{\prime }(e^s_c,r_p,e^o_p)) \end{split}$$   (Eq. 13)
where, $\gamma $ is margin, $e^s_c$ represent corrupted entity and $g^{\prime }(e^s_c,r_p,e^o_p)$ represent corrupted triplet score.
Linkage Learning Loss: We design a novel loss function to leverage pairwise label set $\mathcal {L}$ . Given a triplet $(e^s_X, r_X, e^o_X)$ from knowledge graph $X$ , we first find the entity $e_Y^+$ from graph $Y$ that represent the same real-world entity as $e^s_X$ . We then replace $e^s_X$ with $e_Y^+$ and compute score $g(e_Y^+,r_X,e^o_X)$ . Next, we find set of all entities $E_Y^-$ from graph $(e^s_X, r_X, e^o_X)$0 that has a negative label with entity $(e^s_X, r_X, e^o_X)$1 . We consider them analogous to the negative samples we generated for Eq. 13 . We then propose the label learning loss function as:
$$\begin{split} L_{lab} &= \sum \limits _{z=1}^{Z} \max (0, \gamma - g(e_Y^+,r_X,e^o_X) \\ &+ (g^{\prime }(e_Y^-,r_X,e^o_X)_z)) \end{split}$$   (Eq. 14)
where, $Z$ is the total number of negative labels for $e_X$ . $\gamma $ is margin which is usually set to 1 and $e_Y^- \in E_Y^-$ represent entity from graph $Y$ with which entity $e^s_X$ had a negative label. Please note that this applies symmetrically for the triplets that originate from graph $Y$ in the overall dataset. Note that if both entities of a triplet have labels, we will include both cases when computing the loss. Eq. 14 is inspired by Insight 1 and Insight 2 defined earlier in Section 2. Given a set $\mathcal {D}$ of $N$ observed triplets across two graphs, we define complete multi-task objective as:
$$\mathbf {L}(\mathbf {\Omega }) = \sum \limits _{i=1}^{N} [b \cdot L_{rel} + (1-b) \cdot L_{lab}] + \lambda \left\Vert \mathbf {\Omega } \right\Vert _2^2$$   (Eq. 15)
where $\mathbf {\Omega }$ is set of all model parameters and $\lambda $ is regularization hyper-parameter. $b$ is weight hyper-parameter used to attribute importance to each task. We train with mini-batch SGD procedure (Algorithm "Objective Function" ) using Adam Optimizer. [t!] LinkNBed mini-batch Training Input: Mini-batch $\mathcal {M}$ , Negative Sample Size $C$ , Negative Label Size $Z$ , Attribute data $att\_data$ , Neighborhood data $nhbr\_data$ , Type data $type\_data$ , Positive Label Dict $pos\_dict$ , Negative Label Dict $\lambda $0 Output: Mini-batch Loss $\lambda $1 . $\lambda $2 score_pos = []; score_neg = []; score_pos_lab = []; score_neg_lab = [] $\lambda $3 to size( $\lambda $4 ) input_tuple = $\lambda $5 = ( $\lambda $6 ) sc = compute_triplet_score( $\lambda $7 ) (Eq. "Relational Score Function" ) score_pos.append(sc) $\lambda $8 to $\lambda $9 Select $b$0 from entity list such that $b$1 and $b$2 and $b$3 sc_neg = compute_triplet_score( $b$4 ) score_neg.append(sc_neg) $b$5 in $b$6 $b$7 = positive label for $b$8 sc_pos_l = compute_triplet_score( $b$9 ) score_pos_lab.append(sc_pos_l) $\mathcal {M}$0 to $\mathcal {M}$1 Select $\mathcal {M}$2 from $\mathcal {M}$3 sc_neg_l = compute_triplet_score( $\mathcal {M}$4 ) score_neg_lab.append(sc_neg_l) $\mathcal {M}$5 compute_minibatch_loss(score_pos, score_neg, score_pos_lab, score_neg_lab) (Eq. 15 ) Back-propagate errors and update parameters $\mathcal {M}$6 return $\mathcal {M}$7
Missing Positive Labels. It is expensive to obtain positive labels across multiple graphs and hence it is highly likely that many entities will not have positive labels available. For those entities, we will modify Eq. 14 to use the original triplet $(e^s_X, r_X, e^o_X)$ in place of perturbed triplet $g(e_Y^+,r_X,e^o_X)$ for the positive label. The rationale here again arises from Insight 2 wherein embeddings of two duplicate entities should be able to replace each other without affecting the score.
Training Time Complexity. Most contextual information is pre-computed and available to all training steps which leads to constant time embedding lookup for those context. But for attribute network, embedding needs to be computed for each attribute separately and hence the complexity to compute score for one triplet is $\mathcal {O}(2a)$ where $a$ is number of attributes. Also for training, we generate $C$ negative samples for relational loss function and use $Z$ negative labels for label loss function. Let $k = C + Z$ . Hence, the training time complexity for a set of $n$ triplets will be $\mathcal {O}(2ak*n)$ which is linear in number of triplets with a constant factor as $ak << n$ for real world knowledge graphs. This is desirable as the number of triplets tend to be very large per graph in multi-relational settings.
Memory Complexity. We borrow notations from BIBREF9 and describe the parameter complexity of our model in terms of the number of each component and corresponding embedding dimension requirements. Let $H_a = 2*N_eH_e + N_rH_r + N_tH_t + N_kH_k + N_vH_v$ . The parameter complexity of our model is: $H_a * (H_b + 1)$ . Here, $N_e$ , $N_r$ , $N_t$ , $N_k$ , $N_v$ signify number of entities, relations, types, attribute keys and vocab size of attribute values across both datasets. Here $H_b$ is the output dimension of the hidden layer.
Datasets
We evaluate LinkNBed and baselines on two real world knowledge graphs: D-IMDB (derived from large scale IMDB data snapshot) and D-FB (derived from large scale Freebase data snapshot). Table 1 provides statistics for our final dataset used in the experiments. Appendix B.1 provides complete details about dataset processing.
Baselines
We compare the performance of our method against state-of-the-art representation learning baselines that use neural embedding techniques to learn entity and relation representation. Specifically, we consider compositional methods of RESCAL BIBREF10 as basic matrix factorization method, DISTMULT BIBREF14 as simple multiplicative model good for capturing symmetric relationships, and Complex BIBREF11 , an upgrade over DISTMULT that can capture asymmetric relationships using complex valued embeddings. We also compare against translational model of STransE that combined original structured embedding with TransE and has shown state-of-art performance in benchmark testing BIBREF23 . Finally, we compare with GAKE BIBREF24 , a model that captures context in entity and relationship representations.
In addition to the above state-of-art models, we analyze the effectiveness of different components of our model by comparing with various versions that use partial information. Specifically, we report results on following variants:
LinkNBed - Embed Only. Only use entity embeddings, LinkNBed - Attr Only. Only use Attribute Context, LinkNBed - Nhbr Only. Only use Neighborhood Context, LinkNBed - Embed + Attr. Use both Entity embeddings and Attribute Context, LinkNBed - Embed + Nhbr. Use both Entity embeddings and Neighbor Context and LinkNBed - Embed All. Use all three Contexts.
Evaluation Scheme
We evaluate our model using two inference tasks:
Link Prediction. Given a test triplet $(e^s, r, e^o)$ , we first score this triplet using Eq. "Relational Score Function" . We then replace $e^o$ with all other entities in the dataset and filter the resulting set of triplets as shown in BIBREF12 . We score the remaining set of perturbed triplets using Eq. "Relational Score Function" . All the scored triplets are sorted based on the scores and then the rank of the ground truth triplet is used for the evaluation. We use this ranking mechanism to compute HITS@10 (predicted rank $\le $ 10) and reciprocal rank ( $\frac{1}{rank}$ ) of each test triplet. We report the mean over all test samples.
Entity Linkage. In alignment with Insight 2, we pose a novel evaluation scheme to perform entity linkage. Let there be two ground truth test sample triplets: $(e_X, e_Y^+, 1)$ representing a positive duplicate label and $(e_X, e_Y^-, 0)$ representing a negative duplicate label. Algorithm "Evaluation Scheme" outlines the procedure to compute linkage probability or score $q$ ( $ \in [0,1]$ ) for the pair $(e_X, e_Y)$ . We use $L1$ distance between the two vectors analogous to Mean Absolute Error (MAE). In lieu of hard-labeling test pairs, we use score $q$ to compute Area Under the Precision-Recall Curve (AUPRC).
[t!] Entity Linkage Score Computation Input: Test pair – $(e_X \in X, e_Y \in Y)$ . Output: Linkage Score – $q$ . 1. Collect all triplets involving $e_X$ from graph $X$ and all triplets involving $e_Y$ from graph $Y$ into a combined set $\mathcal {O}$ . Let $|\mathcal {O}| = k$ . 2. Construct $S_{orig} \in \mathbb {R}^k$ . For each triplet $o \in \mathcal {O}$ , compute score $q$0 using Eq. "Relational Score Function" and store the score in $q$1 . 3. Create triplet set $q$2 as following: $q$3 contain $q$4 Replace $q$5 with $q$6 to create perturbed triplet $q$7 and store it in $q$8 $q$9 contain $e_X$0 Replace $e_X$1 with $e_X$2 to create perturbed triplet $e_X$3 and store it in $e_X$4 4. Construct $e_X$5 . For each triplet $e_X$6 , compute score $e_X$7 using Eq. "Relational Score Function" and store the score in $e_X$8 . 5. Compute $e_X$9 . Elements in $X$0 and $X$1 have one-one correspondence so take the mean absolute difference: $X$2 = $X$3 - $X$4 return $X$5
For the baselines and the unsupervised version (with no labels for entity linkage) of our model, we use second stage multilayer Neural Network as classifier for evaluating entity linkage. Appendix B.2 provides training configuration details.
Predictive Analysis
Link Prediction Results. We train LinkNBed model jointly across two knowledge graphs and then perform inference over individual graphs to report link prediction reports. For baselines, we train each baseline on individual graphs and use parameters specific to the graph to perform link prediction inference over each individual graph. Table 2 shows link prediction performance for all methods. Our model variant with attention mechanism outperforms all the baselines with $4.15\%$ improvement over single graph state-of-the-art Complex model on D-IMDB and $8.23\%$ improvement on D-FB dataset. D-FB is more challenging dataset to learn as it has a large set of sparse relationships, types and attributes and it has an order of magnitude lesser relational evidence (number of triplets) compared to D-IMDB. Hence, LinkNBed's pronounced improvement on D-FB demonstrates the effectiveness of the model. The simplest version of LinkNBed with only entity embeddings resembles DISTMULT model with different objective function. Hence closer performance of those two models aligns with expected outcome. We observed that the Neighborhood context alone provides only marginal improvements while the model benefits more from the use of attributes. Despite being marginal, attention mechanism also improves accuracy for both datasets. Compared to the baselines which are obtained by trained and evaluated on individual graphs, our superior performance demonstrates the effectiveness of multi-graph learning.
Entity Linkage Results. We report entity linkage results for our method in two settings: a.) Supervised case where we train using both the objective functions. b.) Unsupervised case where we learn with only the relational loss function. The latter case resembles the baseline training where each model is trained separately on two graphs in an unsupervised manner. For performing the entity linkage in unsupervised case for all models, we first train a second stage of simple neural network classifier and then perform inference. In the supervised case, we use Algorithm "Evaluation Scheme" for performing the inference. Table 3 demonstrates the performance of all methods on this task. Our method significantly outperforms all the baselines with $33.86\%$ over second best baseline in supervised case and $17.35\%$ better performance in unsupervised case. The difference in the performance of our method in two cases demonstrate that the two training objectives are helping one another by learning across the graphs. GAKE's superior performance on this task compared to the other state-of-the-art relational baselines shows the importance of using contextual information for entity linkage. Performance of other variants of our model again demonstrate that attribute information is more helpful than neighborhood context and attention provides marginal improvements. We provide further insights with examples and detailed discussion on entity linkage task in Appendix A.
Neural Embedding Methods for Relational Learning
Compositional Models learn representations by various composition operators on entity and relational embeddings. These models are multiplicative in nature and highly expressive but often suffer from scalability issues. Initial models include RESCAL BIBREF10 that uses a relation specific weight matrix to explain triplets via pairwise interactions of latent features, Neural Tensor Network BIBREF20 , more expressive model that combines a standard NN layer with a bilinear tensor layer and BIBREF6 that employs a concatenation-projection method to project entities and relations to lower dimensional space. Later, many sophisticated models (Neural Association Model BIBREF25 , HoLE BIBREF26 ) have been proposed. Path based composition models BIBREF27 and contextual models GAKE BIBREF24 have been recently studied to capture more information from graphs. Recently, model like Complex BIBREF11 and Analogy BIBREF28 have demonstrated state-of-the art performance on relational learning tasks. Translational Models ( BIBREF29 , BIBREF30 , BIBREF12 , BIBREF31 , BIBREF32 , BIBREF13 ) learn representation by employing translational operators on the embeddings and optimizing based on their score. They offer an additive and efficient alternative to expensive multiplicative models. Due to their simplicity, they often loose expressive power. For a comprehensive survey of relational learning methods and empirical comparisons, we refer the readers to BIBREF9 , BIBREF23 , BIBREF33 and BIBREF14 . None of these methods address multi-graph relational learning and cannot be adapted to tasks like entity linkage in straightforward manner.
Entity Resolution in Relational Data
Entity Resolution refers to resolving entities available in knowledge graphs with entity mentions in text. BIBREF34 proposed entity disambiguation method for KB population, BIBREF35 learns entity embeddings for resolution, BIBREF36 propose a sophisticated DNN architecture for resolution, BIBREF37 proposes entity resolution across multiple social domains, BIBREF38 jointly embeds text and knowledge graph to perform resolution while BIBREF39 proposes Attention Mechanism for Collective Entity Resolution.
Learning across multiple graphs
Recently, learning over multiple graphs have gained traction. BIBREF15 divides a multi-relational graph into multiple homogeneous graphs and learns associations across them by employing product operator. Unlike our work, they do not learn across multiple multi-relational graphs. BIBREF40 provides logic based insights for cross learning, BIBREF16 does pairwise entity matching across multi-relational graphs and is very expensive, BIBREF41 learns embeddings to support multi-lingual learning and Big-Align BIBREF17 tackles graph alignment problem efficiently for bipartite graphs. None of these methods learn latent representations or jointly train graph alignment and learning which is the goal of our work.
Concluding Remarks and Future Work
We present a novel relational learning framework that learns entity and relationship embeddings across multiple graphs. The proposed representation learning framework leverage an efficient learning and inference procedure which takes into account the duplicate entities representing the same real-world entity in a multi-graph setting. We demonstrate superior accuracies on link prediction and entity linkage tasks compared to the existing approaches that are trained only on individual graphs. We believe that this work opens a new research direction in joint representation learning over multiple knowledge graphs.
Many data driven organizations such as Google and Microsoft take the approach of constructing a unified super-graph by integrating data from multiple sources. Such unification has shown to significantly help in various applications, such as search, question answering, and personal assistance. To this end, there exists a rich body of work on linking entities and relations, and conflict resolution (e.g., knowledge fusion BIBREF6 . Still, the problem remains challenging for large scale knowledge graphs and this paper proposes a deep learning solution that can play a vital role in this construction process. In real-world setting, we envision our method to be integrated in a large scale system that would include various other components for tasks like conflict resolution, active learning and human-in-loop learning to ensure quality of constructed super-graph. However, we point out that our method is not restricted to such use cases—one can readily apply our method to directly make inference over multiple graphs to support applications like question answering and conversations.
For future work, we would like to extend the current evaluation of our work from a two-graph setting to multiple graphs. A straightforward approach is to create a unified dataset out of more than two graphs by combining set of triplets as described in Section 2, and apply learning and inference on the unified graph without any major change in the methodology. Our inductive framework learns functions to encode contextual information and hence is graph independent. Alternatively, one can develop sophisticated approaches with iterative merging and learning over pairs of graphs until exhausting all graphs in an input collection.
Acknowledgments
We would like to give special thanks to Ben London, Tong Zhao, Arash Einolghozati, Andrew Borthwick and many others at Amazon for helpful comments and discussions. We thank the reviewers for their valuable comments and efforts towards improving our manuscript. This project was supported in part by NSF(IIS-1639792, IIS-1717916).
Discussion and Insights on Entity Linkage Task
Entity linkage task is novel in the space of multi-graph learning and yet has not been tackled by any existing relational learning approaches. Hence we analyze our performance on the task in more detail here. We acknowledge that baseline methods are not tailored to the task of entity linkage and hence their low performance is natural. But we observe that our model performs well even in the unsupervised scenario where essentially the linkage loss function is switched off and our model becomes a relational learning baseline. We believe that the inductive ability of our model and shared parameterization helps to capture knowledge across graphs and allows for better linkage performance. This outcome demonstrates the merit in multi-graph learning for different inference tasks. Having said that, we admit that our results are far from comparable to state-of-the-art linkage results (Das et al., 2017) and much work needs to be done to advance representation and relational learning methods to support effective entity linkage. But we note that our model works for multiple types of entities in a very heterogeneous environment with some promising results which serves as an evidence to pursue this direction for entity linkage task.
We now discuss several use-case scenarios where our model did not perform well to gain insights on what further steps can be pursued to improve over this initial model:
Han Solo with many attributes (False-negative example). Han Solo is a fictional character in Star Wars and appears in both D-IMDB and D-FB records. We have a positive label for this sample but we do not predict it correctly. Our model combines multiple components to effectively learn across graphs. Hence we investigated all the components to check for the failures. One observation we have is the mismatch in the amount of attributes across the two datasets. Further, this is compounded by multi-value attributes. As described, we use paragraph2vec like model to learn attribute embeddings where for each attribute, we aggregate over all its values. This seems to be computing embeddings that are very noisy. As we have seen attributes are affecting the final result with high impact and hence learning very noisy attributes is not helping. Further, the mismatch in number of types is also an issue. Even after filtering the types, the difference is pretty large. Types are also included as attributes and they contribute context to relation embeddings. We believe that the skew in type difference is making the model learn bad embeddings. Specifically this happens in cases where lot of information is available like Han Solo as it lead to the scenario of abundant noisy data. With our investigation, we believe that contextual embeddings need further sophistication to handle such scenarios. Further, as we already learn relation, type and attribute embeddings in addition to entity embeddings, aligning relations, types and attributes as integral task could also be an important future direction.
Alfred Pennyworth is never the subject of matter (False-negative example). In this case, we observe a new pattern which was found in many other examples. While there are many triples available for this character in D-IMDB, very few triplets are available in D-FB. This skew in availability of data hampers the learning of deep network which ends up learning very different embeddings for two realizations. Further, we observe another patter where Alfred Pennyworth appears only as an object in all those few triplets of D-FB while it appears as both subject and object in D-IMDB. Accounting for asymmetric relationships in an explicit manner may become helpful for this scenario.
Thomas Wayne is Martha Wayne! (False-positive example). This is the case of abundance of similar contextual information as our model predicts Thomas Wayne and Martha Wayne to be same entity. Both the characters share a lot of context and hence many triples and attributes, neighborhood etc. are similar for of them eventually learning very similar embeddings. Further as we have seen before, neighborhood has shown to be a weak context which seems to hamper the learning in this case. Finally, the key insight here is to be able to attend to the very few discriminative features for the entities in both datasets (e.g. male vs female) and hence a more sophisticated attention mechanism would help.
In addition to the above specific use cases, we would like to discuss insights on following general concepts that naturally occur when learning over multiple graphs:
Additional Dataset Details
We perform light pre-processing on the dataset to remove self-loops from triples, clean the attributes to remove garbage characters and collapse CVT (Compound Value Types) entities into single triplets. Further we observe that there is big skew in the number of types between D-IMDB and D-FB. D-FB contains many non-informative type information such as $\#base.*$ . We remove all such non-informative types from both datasets which retains 41 types in D-IMDB and 324 types in D-FB. This filtering does not reduce the number of entities or triples by significant number (less than 1000 entities filtered)
For comparing at scale with baselines, we further reduce dataset using similar techniques adopted in producing widely accepted FB-15K or FB-237K. Specifically, we filter relational triples such that both entities in a triple contained in our dataset must appear in more than $k$ triples. We use $k=50$ for D-FB and $k=100$ for D-IMDB as D-IMDB has orders of magnitude more triples compared to D-FB in our curated datasets. We still maintain the overall ratio of the number of triples between the two datasets.
Positive and Negative Labels. We obtain 500662 positive labels using the existing links between the two datasets. Note that any entity can have only one positive label. We also generate 20 negative labels for each entity using the following method: (i) randomly select 10 entities from the other graph such that both entities belong to the same type and there exist no positive label between entities (ii) randomly select 10 entities from the other graph such that both entities belong to different types.
Training Configurations
We performed hyper-parameter grid search to obtain the best performance of our method and finally used the following configuration to obtain the reported results:
– Entity Embedding Size: 256, Relation Embedding Size=64, Attribute Embedding Size = 16, Type Embedding Size = 16, Attribute Value Embedding Size = 512. We tried multiple batch sizes with very minor difference in performance and finally used size of 2000. For hidden units per layer, we use size = 64. We used $C=50$ negative samples and $Z=20$ negative labels. The learning rate was initialized as 0.01 and then decayed over epochs. We ran our experiments for 5 epochs after which the training starts to convert as the dataset is very large. We use loss weights $b$ as 0.6 and margin as 1. Further, we use $K = 50$ random walks of length $l = 3$ for each entity We used a train/test split of 60%/40% for both the triples set and labels set. For baselines, we used the implementations provided by the respective authors and performed grid search for all methods according to their requirements.
Contextual Information Formulations
Here we describe exact formulation of each context that we used in our work.
Neighborhood Context: Given a triplet $(e^s,r,e^o)$ , the neighborhood context for an entity $e^s$ will be all the nodes at 1-hop distance from $e^s$ other than the node $e^o$ . This will capture the effect of other nodes in the graph surrounding $e^s$ that drives $e^s$ to participate in fact $(e^s,r,e^o)$ . Concretely, we define the neighborhood context of $e^s$ as follows:
Nc(es) = 1ne' e' N(es)
e' eo ve'
where $\mathcal {N}(e^s)$ is the set of all entities in neighborhood of $e^s$ other than $e^o$ . We collect the neighborhood set for each entity as a pre-processing step using a random walk method. Specifically, given a node $e$ , we run $k$ rounds of random-walks of length $l$ and create the neighborhood set $\mathcal {N}(e)$ by adding all unique nodes visited across these walks.
Please note that we can also use $\max $ function in ( "Contextual Information Formulations" ) instead of sum. $\mathbf {N_c}(e^s) \in \mathbb {R}^d$ and the context can be similarly computed for object entity.
Attribute Context. For an entity $e^s$ , the corresponding attribute context is defined as
Ac(es) = 1na i=1na aies
where $n_a$ is the number of attributes. $\mathbf {a_i^{e^s}}$ is the embedding for attribute $i$ . $\mathbf {A_c}(e^s) \in \mathbb {R}^y$ .
Type Context. We use type context mainly for relationships i.e. for a given relationship $r$ , this context aims at capturing the effect of type of entities that have participated in this relationship. For a given triplet $(e^s, r , e^o)$ , we define type context for relationship $r$ as:
Tc(r) = 1ntr i=1ntr vit'
where, $n_t^r$ is the total number of types of entities that has participated in relationship $r$ and $\mathbf {v_i^{t^{\prime }}}$ is the type embedding that corresponds to type $t$ . $\mathbf {T_c}(r) \in \mathbb {R}^q$ .

Question: On what data is the model evaluated?
Answer: D-IMDB (derived from large scale IMDB data snapshot), D-FB (derived from large scale Freebase data snapshot)