Question: What are the two extensions they propose to the order-embedding model?
Evidence: A core problem in artificial intelligence is to capture, in machine-usable form, the collection of information that an ordinary person would have, known as commonsense knowledge. For example, a machine should know that a room may have a door, and that when a person enters a room, it is generally through a door. This background knowledge is crucial for solving many difficult, ambiguous natural language problems in coreference resolution and question answering, as well as the creation of other reasoning machines.
More than just curating a static collection of facts, we would like commonsense knowledge to be represented in a way that lends itself to machine reasoning and inference of missing information. We concern ourselves in this paper with the problem of learning commonsense knowledge representations.
In machine learning settings, knowledge is usually represented as a hypergraph of triplets such as Freebase BIBREF1 , WordNet BIBREF2 , and ConceptNet BIBREF3 . In these knowledge graphs, nodes represent entities or terms $t$ , and hyperedges are relations $R$ between these entities or terms, with each fact in the knowledge graph represented as a triplet $<t_1, R, t_2>$ . Researchers have developed many models for knowledge representation and learning in this setting BIBREF4 , BIBREF5 , BIBREF6 , BIBREF7 , BIBREF8 , under the umbrella of knowledge graph completion. However, none of these naturally lend themselves to traditional methods of logical reasoning such as transitivity and negation.
While a knowledge graph completion model can represent relations such as Is-A and entailment, there is no mechanism to ensure that its predictions are internally consistent. For example, if we know that a dog is a mammal, and a pit bull is a dog, we would like the model to also predict that a pit bull is a mammal. These transitive entailment relations describe ontologies of hierarchical data, a key component of commonsense knowledge which we focus on in this work.
Recently, a thread of research on representation learning has aimed to create embedding spaces that automatically enforce consistency in these predictions using the intrinsic geometry of the embedding space BIBREF9 , BIBREF0 , BIBREF10 . In these models, the inferred embedding space creates a globally consistent structured prediction of the ontology, rather than the local relation predictions of previous models.
We focus on the order-embedding model BIBREF0 which was proposed for general hierarchical prediction including multimodal problems such as image captioning. While the original work included results on ontology prediction on WordNet, we focus exclusively on the model's application to commonsense knowledge, with its unique characteristics including complex ordering structure, compositional, multi-word entities, and the wealth of commonsense knowledge to be found in large-scale unstructured text data.
We propose two extensions to the order embedding model. The first augments hierarchical supervision from existing ontologies with non-hierarchical knowledge in the form of raw text. We find incorporating unstructured text brings accuracy from 92.0 to 93.0 on a commonsense dataset containing Is-A relations from ConceptNet and Microsoft Concept Graph (MCG), with larger relative gains from smaller amounts of labeled data.
The second extension uses the complex partial-order structure of real-world ontologies to find long-distance triplet constraints among embeddings which are poorly enforced by the standard pairwise training method. By adding our additional triplet constraints to the baseline order-embedding model, we find performance improves from 90.6 to 91.3 accuracy on the WordNet ontology dataset.
We find that order embeddings' ease of extension, both by incorporating non-ordered data, and additional training constraints derived from the structure of the problem, makes it a promising avenue for the development of further algorithms for automatic learning and jointly consistent prediction of ontologies.
We propose two extensions to the order embedding model. The first augments hierarchical supervision from existing ontologies with non-hierarchical knowledge in the form of raw text. We find incorporating unstructured text brings accuracy from 92.0 to 93.0 on a commonsense dataset containing Is-A relations from ConceptNet and Microsoft Concept Graph (MCG), with larger relative gains from smaller amounts of labeled data. The second extension uses the complex partial-order structure of real-world ontologies to find long-distance triplet constraints among embeddings which are poorly enforced by the standard pairwise training method.
Answer: The first augments hierarchical supervision from existing ontologies with non-hierarchical knowledge in the form of raw text. The second extends the complex partial-order structure of real-world ontologies to find long-distance triplet constraints among embeddings which are poorly enforced by the standard pairwise training method.