Linguistic Embeddings as a Common-Sense Knowledge Repository: Challenges and Opportunities

Many applications of linguistic embedding models rely on their value as pretrained inputs for end-to-end tasks such as dialog modeling , machine translation , or question answering . This position paper presents an alternate paradigm : Rather than using learned embeddings as input features , we instead treat them as a common-sense knowledge repository that can be queried via simple mathematical operations within the embedding space . We show how linear offsets can be used to ( a ) identify an object given its description , ( b ) discover relations of an object given its label , and ( c ) map free-form text to a set of action primitives . Our experiments provide a valuable proof of concept that language-informed common sense reasoning , or ‘ reasoning in the linguistic domain ’ , lies within the grasp of the research community . In order to attain this goal , however , we must reconsider the way neural embedding models are typically trained an evaluated . To that end , we also identify three empirically-motivated evaluation metrics for use in the training of future embedding models . 1 INTRODUCTION . This position paper casts pre-trained embedding models like BERT ( Devlin et al. , 2018 ) , GPT-2 ( Radford et al . ) and InferSent ( Conneau et al. , 2017 ) in a new role . Rather than using the learned hidden states as pre-trained features for downstream tasks , we instead view them as a form of emergent , flexible knowledge representation harvested from a rich body of text corpora . We show how the knowledge implicitly encoded in the embedding space can be extracted and utilized to solve real-world problems with little or no additional training , and we argue that the effectiveness of this method can be vastly increased in future embedding models . There is , of course , inherent unpredictability of this approach . In contrast to hand-curated symbolic knowledge , harvested representational knowledge is imprecise , at times unreliable , and difficult to anticipate – but it is also fascinating , spontaneous , and fluidly creative in ways that traditional knowledge systems seldom replicate . Unlike a knowledge graph , these vector-based representations can explore questions like , “ What is the combination of fear and sound ? ” or “ What is like a river without water ? 1 ” Queries , in this context , are linear algebra operations on the embedded representations of input text , with answers found by seeking the embedded word or sentence with the greatest cosign similarity to the calculated result . This requires a startlingly high level of semantic structure within the geometry of the learned representations , and gentle probing of current state-of-the-art embedding spaces reveals that they are not yet up to the task . The potential exists , but only in nascent form . The purpose of this paper is to unequivocally demonstrate that potential , quantify it where possible , and propose new evaluation metrics for the development of embedding models with optimal semantic geometries . 1If you query a FastText ( Bojanowski et al. , 2017 ) model , the answers are “ scream ” and “ road ” . 2 RELATED WORK . Language-based knowledge representation falls roughly into two categories . The first includes knowledge graphs ( Vrandecic & Krotzsch , 2014 ; Liu & Singh , 2004 ; Saxena et al. , 2014 ) , lexical databases ( Miller , 1995 ) , and structured ontologies ( Matuszek et al. , 2006 ) . Each of these has distinct characteristics , but they share the principle of representing ideas as distinct entities linked by conceptual relations . Lexical knowledge may be hand-crafted ( Liu & Singh , 2004 ; Miller , 1995 ; Matuszek et al. , 2006 ) or extracted from the internet ( Vrandecic & Krotzsch , 2014 ; Saxena et al. , 2014 ) , although almost all such extracted knowledge bases involve some form of human curation . The second category is model-based , and involves allowing the system to learn a numerical representation for each word or phrase encountered . A neural network or statistical model is commonly used for this purpose , with the resulting representations interchangeably referred to as vector spaces ( Mikolov et al. , 2013a ; Turney & Pantel , 2010 ) , embedding spaces ( Liu et al. , 2015 ; Fulda et al. , 2017a ; Bolukbasi et al. , 2016 ) , or distributed representations of words ( Le & Mikolov , 2014a ; Mikolov et al. , 2013b ; Le & Mikolov , 2014b ) . While these models are heavily valued for their use as input features for downstream tasks , the information encoded within the embedding space is seldom directly extracted , particularly when the model encodes entire sentences or multi-word phrases . One purpose of this paper is to raise a warning voice : In the mad scramble to train more clever and more effective embedding models , we should take note of our assumptions about what a ‘ good ’ embedding model looks like . One common evaluation method for neural language models is to measure their effectiveness as pre-trained input features for downstream tasks . Bert ( Devlin et al. , 2018 ) , InferSent ( Conneau et al. , 2017 ) and Google ’ s Universal Sentence Encoder ( Cer et al. , 2018a ) were all evaluated using this method . However , while this evaluation method can reveal how well a system is learning language data in a general sense , it can not tell us whether the resultant embedding space exhibits strong semantic structure . After all , a model trained to predict semantic similarity is not the same as a model whose sentence representations embody semantic similarity . We therefore caution readers that currently popular evaluation metrics for linguistic embeddings may be overlooking some of the models ’ greatest potential strengths , and explicitly encourage efforts like those of Conneau et al . ( Conneau et al. , 2018 ) and zhu et al . ( Zhu et al. , 2018 ) to examine the semantic properties of learned representations directly . Otherwise , we may inadvertently bias our research agendas away from the very properties that make these embedding spaces so delightfully intriguing . 3 BACKGROUND : SEMANTIC PROPERTIES OF LEARNED REPRESENTATIONS . Word-level embedding spaces exhibit intriguing semantic geometries . For example , it is possible to perform vector operations within the embedding space to answer queries such as Spain : Madrid : :France : ? , where the unknown value ? should be replaced by Paris ( Mikolov et al. , 2013b ; c ) . Similarly , targeted projections of the embedding space onto hand-selected basis vectors reveal that words that describe common household objects are positioned in close proximity to objects that tend to appear within the same room . For example , common appliances like ‘ stove ’ , ‘ refrigerator ’ , and ‘ blender ’ lie close to one another and share a common angle of incidence toward the representation for ‘ kitchen ’ ( see Figure 1 ) . These observations suggest that a form of everyday common-sense knowledge is implicitly encoded within the structure of the embedding space . It ‘ knows ’ , in some sense , that Paris is a city in France , just as it ‘ knows ’ that an oven can be found in a kitchen . Further demonstrations of this principle include the successful use of tagged word2vec embeddings to reason about the affordances of physical objects ( Fulda et al. , 2017a ) and to determine the best mode of travel for reaching vacation destinations ( Fulda et al. , 2017b ) . Unfortunately , the structural properties of word-level embedding spaces fail to manifest effectively in their sentence-level extensions . For example , most state-of-the-art embedding models map negation pairs such as “ I am a cat ” and “ I am not a cat ” to nearly identical vectors ( see Figure 2 ) . The models were not optimized for lingusitic reasoning tasks , and their structural irregularities prevent them from being easily used in the same way that word embeddings are applied . Perhaps for this reason , many applications of pre-trained embeddings focus on their potential for transfer learning rather than on attempts to extract information directly from the embedded representation . A major contribution of this paper is the idea that common-sense knowledge can be extracted directly from multi-word linguistic embeddings , and that this knowledge can be utilized for real-world applications . This is partially due to the improved structural properties of recently-developed embedding spaces , partially due to an extraction mechanism that focuses on the relative positions of sentences rather than on their absolute locations , and partially due to the careful selection of reasoning tasks that are compatible with the learned geometries of modern embedding spaces . 4 INDEXING : IDENTIFYING AN OBJECT FROM ITS DESCRIPTION . When dealing with knowledge systems , a first and critical step involves identifying an entry point : In other words , which node should the system examine in order to access knowledge relevant to its current situation ? In this section , we show that linguistic embeddings can be used to identify the object referred to by a free-form description , often with surprising accuracy . A key observation here is that different models not only perform differently , but they each answer a different subset of queries correctly . This suggests ( but certainly does not prove ) that a particularly well-formulated embedding space might facilitate much higher accuracies , if such an embedding space can be properly trained . 4.1 METHODOLOGY . Given a free-form text description x , a set of possible node labels L = { l1 , ... , ln } , and two sets of example texts A = { A1 , ... , An } and B = { B1 , ... , Bn } , where each Ai is a free-form description of some object , and each Bi is the object so described , a linguistic embedding model can be queried for common-sense knowledge in the following manner : 1 . Use the model to convert x into a vector representation v 2 . Create a canonical vector V = 1n ∑ i ( Bi −Ai ) 3 . Apply the canonical vector to obtain an indexing point p = v + V 4 . Search the set of possible node labels to find l∗ = argminl∈L distcos ( l , p ) Here , distcos is the cosine distance between two vectors . 4.2 INDEXING CASE STUDY Figure 4 shows the indexes selected by six neural embedding models when given the texts in Figure 3 as canonical examples and twelve node labels to choose from . We begin by noting that we are using these embedding models in ways for which they were not designed . GPT-2 and Transformer-XL , in particular , were trained not as general purpose sentence representations but for the very specific purpose of language modeling and text generation . It is perhaps not surprising , then , that they are not well suited to the task of node indexing via vector offset methods . Nevertheless we include them in this study because it is enlightening to observe that different training corpora and model architectures create a startling amount of variance in the semantic geometry of the resultant embedding space . While a case study of this size is far from conclusive , the results tell an interesting story . Of the embedding models examined , Google ’ s Universal Sentence Encoder ( lite ) and Facebook ’ s InferSent model clearly have the most effective geometries for this task . Both models were able to index 75 % of the free-form text descriptions correctly , and even when they selected an incorrect label , they usually selected it from the right general category of answers . We find this impressive and evocative of future possibilities . The question quickly arises : If models that were trained without this specific application in mind can accomplish an indexing task with reasonable accuracy , what might a model accomplish if it were trained with the specific objective of facilitating knowledge retrieval via vector offset methods ? Further thoughts in this direction are outlined in Section 7 .

Summary: - This paper claims to provide a “alternate” view of pretrained embedding as commonsense repository. But this is not a new view at all. It is well known embedding can encode commonsense knowledge, which is a reason it helps a wide variety of recent commonsense related tasks (and there has been considerable analysis on what kind of coomonsense is learned and helpful).