Linguistic Embeddings as a Common-Sense Knowledge Repository: Challenges and Opportunities

Many applications of linguistic embedding models rely on their value as pretrained inputs for end-to-end tasks such as dialog modeling , machine translation , or question answering . This position paper presents an alternate paradigm : Rather than using learned embeddings as input features , we instead treat them as a common-sense knowledge repository that can be queried via simple mathematical operations within the embedding space . We show how linear offsets can be used to ( a ) identify an object given its description , ( b ) discover relations of an object given its label , and ( c ) map free-form text to a set of action primitives . Our experiments provide a valuable proof of concept that language-informed common sense reasoning , or ‘ reasoning in the linguistic domain ’ , lies within the grasp of the research community . In order to attain this goal , however , we must reconsider the way neural embedding models are typically trained an evaluated . To that end , we also identify three empirically-motivated evaluation metrics for use in the training of future embedding models . 1 INTRODUCTION . This position paper casts pre-trained embedding models like BERT ( Devlin et al. , 2018 ) , GPT-2 ( Radford et al . ) and InferSent ( Conneau et al. , 2017 ) in a new role . Rather than using the learned hidden states as pre-trained features for downstream tasks , we instead view them as a form of emergent , flexible knowledge representation harvested from a rich body of text corpora . We show how the knowledge implicitly encoded in the embedding space can be extracted and utilized to solve real-world problems with little or no additional training , and we argue that the effectiveness of this method can be vastly increased in future embedding models . There is , of course , inherent unpredictability of this approach . In contrast to hand-curated symbolic knowledge , harvested representational knowledge is imprecise , at times unreliable , and difficult to anticipate – but it is also fascinating , spontaneous , and fluidly creative in ways that traditional knowledge systems seldom replicate . Unlike a knowledge graph , these vector-based representations can explore questions like , “ What is the combination of fear and sound ? ” or “ What is like a river without water ? 1 ” Queries , in this context , are linear algebra operations on the embedded representations of input text , with answers found by seeking the embedded word or sentence with the greatest cosign similarity to the calculated result . This requires a startlingly high level of semantic structure within the geometry of the learned representations , and gentle probing of current state-of-the-art embedding spaces reveals that they are not yet up to the task . The potential exists , but only in nascent form . The purpose of this paper is to unequivocally demonstrate that potential , quantify it where possible , and propose new evaluation metrics for the development of embedding models with optimal semantic geometries . 1If you query a FastText ( Bojanowski et al. , 2017 ) model , the answers are “ scream ” and “ road ” . 2 RELATED WORK . Language-based knowledge representation falls roughly into two categories . The first includes knowledge graphs ( Vrandecic & Krotzsch , 2014 ; Liu & Singh , 2004 ; Saxena et al. , 2014 ) , lexical databases ( Miller , 1995 ) , and structured ontologies ( Matuszek et al. , 2006 ) . Each of these has distinct characteristics , but they share the principle of representing ideas as distinct entities linked by conceptual relations . Lexical knowledge may be hand-crafted ( Liu & Singh , 2004 ; Miller , 1995 ; Matuszek et al. , 2006 ) or extracted from the internet ( Vrandecic & Krotzsch , 2014 ; Saxena et al. , 2014 ) , although almost all such extracted knowledge bases involve some form of human curation . The second category is model-based , and involves allowing the system to learn a numerical representation for each word or phrase encountered . A neural network or statistical model is commonly used for this purpose , with the resulting representations interchangeably referred to as vector spaces ( Mikolov et al. , 2013a ; Turney & Pantel , 2010 ) , embedding spaces ( Liu et al. , 2015 ; Fulda et al. , 2017a ; Bolukbasi et al. , 2016 ) , or distributed representations of words ( Le & Mikolov , 2014a ; Mikolov et al. , 2013b ; Le & Mikolov , 2014b ) . While these models are heavily valued for their use as input features for downstream tasks , the information encoded within the embedding space is seldom directly extracted , particularly when the model encodes entire sentences or multi-word phrases . One purpose of this paper is to raise a warning voice : In the mad scramble to train more clever and more effective embedding models , we should take note of our assumptions about what a ‘ good ’ embedding model looks like . One common evaluation method for neural language models is to measure their effectiveness as pre-trained input features for downstream tasks . Bert ( Devlin et al. , 2018 ) , InferSent ( Conneau et al. , 2017 ) and Google ’ s Universal Sentence Encoder ( Cer et al. , 2018a ) were all evaluated using this method . However , while this evaluation method can reveal how well a system is learning language data in a general sense , it can not tell us whether the resultant embedding space exhibits strong semantic structure . After all , a model trained to predict semantic similarity is not the same as a model whose sentence representations embody semantic similarity . We therefore caution readers that currently popular evaluation metrics for linguistic embeddings may be overlooking some of the models ’ greatest potential strengths , and explicitly encourage efforts like those of Conneau et al . ( Conneau et al. , 2018 ) and zhu et al . ( Zhu et al. , 2018 ) to examine the semantic properties of learned representations directly . Otherwise , we may inadvertently bias our research agendas away from the very properties that make these embedding spaces so delightfully intriguing . 3 BACKGROUND : SEMANTIC PROPERTIES OF LEARNED REPRESENTATIONS . Word-level embedding spaces exhibit intriguing semantic geometries . For example , it is possible to perform vector operations within the embedding space to answer queries such as Spain : Madrid : :France : ? , where the unknown value ? should be replaced by Paris ( Mikolov et al. , 2013b ; c ) . Similarly , targeted projections of the embedding space onto hand-selected basis vectors reveal that words that describe common household objects are positioned in close proximity to objects that tend to appear within the same room . For example , common appliances like ‘ stove ’ , ‘ refrigerator ’ , and ‘ blender ’ lie close to one another and share a common angle of incidence toward the representation for ‘ kitchen ’ ( see Figure 1 ) . These observations suggest that a form of everyday common-sense knowledge is implicitly encoded within the structure of the embedding space . It ‘ knows ’ , in some sense , that Paris is a city in France , just as it ‘ knows ’ that an oven can be found in a kitchen . Further demonstrations of this principle include the successful use of tagged word2vec embeddings to reason about the affordances of physical objects ( Fulda et al. , 2017a ) and to determine the best mode of travel for reaching vacation destinations ( Fulda et al. , 2017b ) . Unfortunately , the structural properties of word-level embedding spaces fail to manifest effectively in their sentence-level extensions . For example , most state-of-the-art embedding models map negation pairs such as “ I am a cat ” and “ I am not a cat ” to nearly identical vectors ( see Figure 2 ) . The models were not optimized for lingusitic reasoning tasks , and their structural irregularities prevent them from being easily used in the same way that word embeddings are applied . Perhaps for this reason , many applications of pre-trained embeddings focus on their potential for transfer learning rather than on attempts to extract information directly from the embedded representation . A major contribution of this paper is the idea that common-sense knowledge can be extracted directly from multi-word linguistic embeddings , and that this knowledge can be utilized for real-world applications . This is partially due to the improved structural properties of recently-developed embedding spaces , partially due to an extraction mechanism that focuses on the relative positions of sentences rather than on their absolute locations , and partially due to the careful selection of reasoning tasks that are compatible with the learned geometries of modern embedding spaces . 4 INDEXING : IDENTIFYING AN OBJECT FROM ITS DESCRIPTION . When dealing with knowledge systems , a first and critical step involves identifying an entry point : In other words , which node should the system examine in order to access knowledge relevant to its current situation ? In this section , we show that linguistic embeddings can be used to identify the object referred to by a free-form description , often with surprising accuracy . A key observation here is that different models not only perform differently , but they each answer a different subset of queries correctly . This suggests ( but certainly does not prove ) that a particularly well-formulated embedding space might facilitate much higher accuracies , if such an embedding space can be properly trained . 4.1 METHODOLOGY . Given a free-form text description x , a set of possible node labels L = { l1 , ... , ln } , and two sets of example texts A = { A1 , ... , An } and B = { B1 , ... , Bn } , where each Ai is a free-form description of some object , and each Bi is the object so described , a linguistic embedding model can be queried for common-sense knowledge in the following manner : 1 . Use the model to convert x into a vector representation v 2 . Create a canonical vector V = 1n ∑ i ( Bi −Ai ) 3 . Apply the canonical vector to obtain an indexing point p = v + V 4 . Search the set of possible node labels to find l∗ = argminl∈L distcos ( l , p ) Here , distcos is the cosine distance between two vectors . 4.2 INDEXING CASE STUDY Figure 4 shows the indexes selected by six neural embedding models when given the texts in Figure 3 as canonical examples and twelve node labels to choose from . We begin by noting that we are using these embedding models in ways for which they were not designed . GPT-2 and Transformer-XL , in particular , were trained not as general purpose sentence representations but for the very specific purpose of language modeling and text generation . It is perhaps not surprising , then , that they are not well suited to the task of node indexing via vector offset methods . Nevertheless we include them in this study because it is enlightening to observe that different training corpora and model architectures create a startling amount of variance in the semantic geometry of the resultant embedding space . While a case study of this size is far from conclusive , the results tell an interesting story . Of the embedding models examined , Google ’ s Universal Sentence Encoder ( lite ) and Facebook ’ s InferSent model clearly have the most effective geometries for this task . Both models were able to index 75 % of the free-form text descriptions correctly , and even when they selected an incorrect label , they usually selected it from the right general category of answers . We find this impressive and evocative of future possibilities . The question quickly arises : If models that were trained without this specific application in mind can accomplish an indexing task with reasonable accuracy , what might a model accomplish if it were trained with the specific objective of facilitating knowledge retrieval via vector offset methods ? Further thoughts in this direction are outlined in Section 7 .

Summary: This paper provides an overview of methods to use embeddings for texts as common-sense knowledge. It mentions many aspects where embeddings can be used as common-sense knowledge. However, the paper lacks both novelty and in-depth analysis.  Most methods proposed are basically computing the cosine distances between language embeddings pairs and find the closest one. It's hard to imagine how to scale up this process in real applications. And most of the analyses are based on cherry-picked examples rather than evaluation with quantitative metrics.