Question: How does Dolores work?
Evidence: Knowledge graphs BIBREF0 enable structured access to world knowledge and form a key component of several applications like search engines, question answering systems and conversational assistants. Knowledge graphs are typically interpreted as comprising of discrete triples of the form (entityA, relationX, entityB) thus representing a relation (relationX) between entityA and entityB. However, one limitation of only a discrete representation of triples is that it does not easily enable one to infer similarities and potential relations among entities which may be missing in the knowledge graph. Consequently, one popular alternative is to learn dense continuous representations of entities and relations by embedding them in latent continuous vector spaces, while seeking to model the inherent structure of the knowledge graph. Most knowledge graph embedding methods can be classified into two major classes: one class which operates purely on triples like RESCAL BIBREF1 , TransE BIBREF2 , DistMult BIBREF3 , TransD BIBREF4 , ComplEx BIBREF5 , ConvE BIBREF6 and the second class which seeks to incorporate additional information (like multi-hops) BIBREF7 . Learning high-quality knowledge graph embeddings can be quite challenging given that (a) they need to effectively model the contextual usages of entities and relations (b) they would need to be useful for a variety of predictive tasks on knowledge graphs.
In this paper, we present a new type of knowledge graph embeddings called Dolores that are both deep and contextualized. Dolores learns both context-independent and context-dependent embeddings of entities and relations through a deep neural sequential model. Figure 1 illustrates the deep contextualized representations learned. Note that the contextually independent entity embeddings (see Figure 1 ) reveal three clusters of entities: writers, philosophers, and musicians. The contextual dependent embeddings in turn effectively account for specific relations. In particular, the context-dependent representations under the relation nationality now nicely cluster the above entities by nationality namely Austrians, Germans, and British/Irish. Similarly Figure 1 shows contextual embeddings given the relation place-lived. Note that these embeddings correctly capture that even though Beethoven and Brahms being Germans, they lived in Vienna and are closer to other Austrian musicians like Schubert.
Unlike most knowledge graph embeddings like TransD, TransE BIBREF2 , BIBREF4 etc. which are typically learned using shallow models, the representations learned by Dolores are deep: dependent on an entire path (rather than just a triple), are functions of internal states of a Bi-Directional LSTM and composed of representations learned at various layers potentially capturing varying degrees of abstractions. Dolores is inspired by recent advances in learning word representations (word embeddings) from deep neural language models using Bi-Directional LSTMs BIBREF8 . In particular, we derive connections between the work of Peters et al. ( BIBREF8 ) who learn deep contextualized word embeddings from sentences using a Bi-Directional LSTM based language model and random walks on knowledge graphs. These connections enable us to propose new “deep contextualized” knowledge graph embeddings which we call Dolores embeddings.
Knowledge Embeddings learned using Dolores can easily be used as input representations for predictive models on knowledge graphs. More importantly, when existing predictive models use input representations for entities and relations, we can easily replace those representations with Dolores representations and significantly improve the performance of existing models. Specifically, we show that Dolores embeddings advance the state-of-the-art models on various tasks like link prediction, triple classification and missing relation type prediction.
To summarize, our contributions are as follows:
Dolores learns both context-independent and context-dependent embeddings of entities and relations through a deep neural sequential model.
Answer: Dolores learns both context-independent and context-dependent embeddings of entities and relations through a deep neural sequential model.