Linguistic Embeddings as a Common-Sense Knowledge Repository: Challenges and Opportunities
Authors: Nancy Fulda
Venue:  (2020)
Keywords: knowledge representation
word embeddings
sentence embeddings
common-sense knowledge

Abstract: Many applications of linguistic embedding models rely on their value as pre-trained inputs for end-to-end tasks such as dialog modeling, machine translation, or question answering. This position paper presents an alternate paradigm: Rather than using learned embeddings as input features, we instead treat them as a common-sense knowledge repository that can be queried via simple mathematical operations within the embedding space. We show how linear offsets can be used to (a) identify an object given its description, (b) discover relations of an object given its label, and (c) map free-form text to a set of action primitives. Our experiments provide a valuable proof of concept that language-informed common sense reasoning, or `reasoning in the linguistic domain', lies within the grasp of the research community. In order to attain this goal, however, we must reconsider the way neural embedding models are typically trained an evaluated. To that end, we also identify three empirically-motivated evaluation metrics for use in the training of future embedding models.

Under review as a conference paper at ICLR 2020
LINGUISTIC EMBEDDINGS AS A COMMON-SENSE KNOWLEDGE REPOSITORY: CHALLENGES AND OP- PORTUNITIES
Anonymous authors Paper under double-bling review
ABSTRACT
Many applications of linguistic embedding models rely their value as pre- trained inputs for end-to-end tasks such as dialog modeling, machine transla- tion, question answering. This position paper presents alternate paradigm: Rather than using learned embeddings as input features, we instead treat them as a common-sense knowledge repository that be queried via simple mathematical operations within the embedding space. We show how linear offsets can be used (a) identify object given its description, (b) discover relations of object given its label, and (c) map free-form text to set of action primitives. Our experi- ments provide valuable proof of concept that language-informel common sense reasoning, or 'reasoning in the linguistic domain', lies within the grasp of the re- search community. In order attain this goal, however, we must reconsider the way neural embedding models are typically trained an evaluated. To that end, we also identify three ampirically-moivatte evaluation metrics for use in the training of future embedding models.
INTRODUCTION
This position paper casts pre-trained embedding models like BERT (Devlin et al., 2018), GPT-2 (Radford et al.) and InferSent (Conneau et al. 2017) in new role. Rather than using the learned hidden states as pre-trained features for downstream tasks, we instead view them as a form of emer- gent, flexible knowledge representation harvested from rich body of text corpora. We show how the knowledge implicitly encoded in the embedding space can be extracted and utilized to solve real-world problems with little or no additional training, and we argue that the effectiveness of this method can be vastly increased in future embedding models.
There is, of course, inherent unpredictability of this approach. contrast to hand-curated symbolic knowledge, harvested representational knowledge is imprecise, at times unreliable, and difficult to anticipate- but also fascinating, spontaneous, and fluidly creative in ways that traditional knowledge systems seldom replicate. Unlike knowledge graph, these vector-based representations can explore questions like, "What is the combination of fear and sound?" or "What is like a river without water
Queries, in this context, are linear algebra operations on the embedded representations of input text, with answers found by seeking the embedded word sentence with the greatest cosign similarity to the calculated result. This requires a startlingly high level of semantic structure within the geom- etry of the learned representations, and gentle probing of current state-of-the-art embedding spaces reveals that they are not yet up to the task. The potential exists, but only in nascent form. The purpose of this paper is unequivocally demonstrate that potential, quantify it where possible, and propose new evaluation metrics for the development of embedding models with optimal semantic geometries.
If you query FastText (Bojanowski al. 2017 model, the answers "seream" and "road".

---

Under review as a conference paper at ICLR 2020
2
RELATED WORK
Language-based knowledge representation falls roughly into two categories. The first includes knowledge graphs ( /Vrandecic & Krotzsch, 2014 Liu & Singh 2004; Saxena et al.) 2014). lexi- cal databases (Miller 1995). and structured ontologies (Matuszek al. 2006). Each of these has distinct characteristics, but they share the principle of representing ideas as distinct entities linked by conceptual relations. Lexical knowledge may be hand-crafted (Liu & Singh 2004, Miller 1995
Matuszek et al. 2006) or extracted from the internet (Vrandecic & Krotzsch 2014 Saxena et al., 2014), although almost all such extracted knowledge bases involve some form of human curation.
The second category is model-based, and involves allowing the system to learn numerical rep- resentation for each word or phrase encountered. A neural network or statistical model com- monly used for this purpose, with the resulting representations interchangeably referred to as vector spaces (Mikolov et al.] 2013a Turney & Pantel, 2010), embedding spaces (Liu et al. 2015 Fulda et al. 2017a Bolukbasi et al.] 2016), or distributed representations of words (Le & Mikolov 2014 Mikoloy et al. 2013b; Le & Mikolov] 2014b). While these models are heavily valued for their use as input features for downstream tasks, the information encoded within the embedding space is seldom directly extracted, particularly when the model encodes entire sentences or multi-word phrases.
One purpose of this paper is to raise a warning voice: In the mad scramble train more clever and more effective embedding models, we should take note of our assumptions about what a 'good' embedding model looks like. One common evaluation method for neural language models is to measure their effectiveness as pre-trained input features for downstream tasks. Bert (Devlin et al., 2018 InferSent (Conneau al., 2017) and Google's 's Universal Sentence Encoder (Cer et al. 201 Sap were all evaluated using this method. However, while this evaluation method can reveal how well system is learning language data in general sense, cannot tell us whether the resultant embedding space exhibits strong semantic structure. After all, model trained to predict semantic similarity is not the same as a model whose sentence representations embody semantic similarity.
We therefore caution readers that currently popular evaluation metrics for linguistic embeddings may be overlooking some of the models' greatest potential strengths, and explicitly encourage efforts like those of Conneau et al. (Conneau et al. 2018) and zhu et al. (Zhu et al.) 2018) to examine the semantic properties of learned representations directly. Otherwise, we may inadvertently bias our research agendas away from the very properties that make these embedding spaces so delightfully intriguing.
BACKGROUND: SEMANTIC PROPERTIES OF LEARNED REPRESENTATIONS
Word-level embedding spaces exhibit intriguing semantic geometries. For example, it is pos- sible to perform vector operations within the embedding space to answer queries such as Spain:Madri:::Fraacc::: where the unknown value should be replaced by Paris (Mikolov et al.) c) Similarly, targeted projections of the embedding space onto hand-selected basis vectors reveal that words that describe common household objects are positioned in close proximity to ob- jects that tend to appear within the same room. For example, common appliances like 'stove", 'refrigerator', and 'blender' lie close to one another and share common angle of incidence toward the representation for 'kitchen' (see Figure I)
These observations suggest that form of everyday common- sense knowledge is implicitly encoded within the structure of the embedding space. It 'knows', in some sense, that Paris is city in France, just it "knows" that an oven can be found in a kitchen. Further demonstrations of this princi- ple include the successful use of tagged word2vee embeddings to reason about the affordances of physical objects (Fulda et al.] (017a) and to determine the best mode of travel for reaching vacation destinations (Fulda et al. 2017b).
Unfortunately, the structural properties of word-level embedding spaces fail to manifest effectively in their sentence-level extensions. For example, most state-of-the of-the-art embedding models map nega- tion pairs such as "I am a cat" and "I am not cat" to nearly identical vectors (see Figure 2). The models were not optimized for lingusitic reasoning tasks, and their structural irregularities prevent them from being easily used in the same way that word embeddings are applied. Perhaps for this
2

---

Under review as a conference paper at ICLR 2020
vectort vectorf vectorrctooooo vwctor/baatrroom baahroomm)
dayroom
gove
eeen
gefngerator
redroom
Jootherush
becttuub
dallway
gothroom
-0.6
-0.4
-0.2
0.0
0.2
0.4
vector| kitchen)) - vecter(refiiggeatrr)
Figure 1: 100-dimensional word2vec embeddings projected into a 2-dimensional space demonstrate impressive semantic structure. Common household items are clustered in specific relationships to the rooms in which they tend to be located. The angle of correspondence is at least as significant as proximity. Of particular interest is the word 'mirror', which is aligned both with the items found in a bedroom and with the items found in a bathroom.
Skip-Thouse
USE lite
BERT
GPT-2
InferSent
Transformer-Xi.
I am cat am not cat am a cat
0.069
0.069
0,070
0.0007
0.241
0.1110
I am a domesticated cat
0.139
0.098
0.188
0.0015
0.147
0.0925
Figure 2: Cosine distances between sentences embedded using various neural models. Each sen- tence pair comprises rows, with distances shown after the second component sentence. Most embedding models place the syntactically similar but semantically distinct sentences 'I am a cat' and 'I a cat' startlingly close one another. Only the recently-develoedd embedding models InferSent and Transformer-Xx (bolded) embed these sentences to positions that are farther apart than the semantically similar sentences 'I a cat' and 'I a domesticated cat'.
reason, many applications of pre-trained embeddings focus their potential for transfer learning rather than on attempts to extract information directly from the embedded representation.
A major contribution of this paper is the idea that common-senss knowledge be extracted di- rectly from multi-word linguistic embeddings, and that this knowledge can be utilized for real-world applications. This is partially due the improved structural properties of recently-develope em- bedding spaces, partially due to an extraction mechanism that focuses on the relative positions of sentences rather than on their absolute locations, and partially due to the careful selection of reason- ing tasks that are compatible with the learned geometries of modern embedding spaces.
4 INDEXING: IDENTIFYING AN OBJECT FROM ITS DESCRIPTION
When dealing with knowledge systems, first and critical step involves identifying an entry point: other words, which node should the system examine in order to access knowledge relevant to its current situation? In this section, we show that linguistic embeddings can be used to identify the ob- ject referred to by a fire--form description, often with surprising accuracy. A key observation here is that different models not only perform differently, but they each answer different subset of queries correctly. This suggests (but certainly does not prove) that a particularly well-formulated embedding space might facilitate much higher accuracies, if such embedding space be properly trained.
3

---

Under review as a conference paper at ICLR 2020
Thop-hh
USk: line
BERT
GPT-2
InferSent
Transt- XL.
trnk and sold rool
coupe
convertible
coupe
converable
soupe
convertible minivan
convertible
coupe
family cur with sliding doors and large cargo area
couge
suy
hollywood
suv
converuble suy
has more ground clearance for ottruad travel
coupe
suy chicago
suv
hollywood
suy
that that's always getling destruyed by monsters
turtorise
holly
hullywood
the place
the
hamburg
wood
suy
bolly
holly wood
purt city in Germany
tokyo
hamburg
hamburg
dack
hamburg
suV
the windy city
coupe
chicago
chicago
holywood
chicago
turtle that walks land
tortodoe
tortodse
dack
tortodse
squat that quacks
turooise
duck
duck
duck
cross between a hosse and a dankey immature cat
tortosse
manvvan
mule
dack
mule
mule
lertorse
kättern
minivan
kitten
kitten
Figure 4: Indexing labels chosen by six neural embedding models when applying the methodology described in Section 4.1] Correct answers are shown in bold-face type. Interestingly. the best per- forming models tend to select label in the right general category even when they do not select the optimal answer.
4.1 METHODOLOGY
Given free-form text description x, of possible node labels T. = (4,:-.,7n) and two of example texts A = {A1. .... A and B = {B...... B, where each A, is a free-form description of some object, and each B; is the object so described, linguistic embedding model can be queried for common-sense knowledge in the following manner:
1. Use the model to convert x into a vector representation
2. Create a canonical vector V Σ, A;)
3. Apply the canonical vector to obtain an indexing p=
4. Search the set of possible node labels to find 1" argminget disteos(l.p)
Here, distcos is the cosine distance between vectors.
4.2 INDEXING CASE STUDY
Figure shows the indexes selected by six neural embedding models when given the texts in Figure
object parrot
description
3 as canonical examples and twelve node labels to
a brighly colored tropical bird that can learn to speak a baby dog with four doors and tra- ditional trunk a vehicle with an open cargo bed in the rear
choose from.
puppy sedan
We begin by noting that we are using these embed- ding models in ways for which they were not de- signed. GPT-2 and Transformer-XL, in particular, were trained not as general purpose sentence repre- sentations but for the very specific purpose of lan- guage modeling and text generation. It is perhaps not surprising, then, that they are not well suited the task of node indexing via vector offset methods. Nevertheless we include them in this study because it is enlightening to observe that different training corpora and model architectures create a startling amount of variance in the semantic geometry of the resultant embedding space.
truck
paris london
the city where the eifel tower is capital city of great britain
Figure 3: Object labels A; and descriptions B, that define the canonical vector V/ used in the experiments for Figure
While a case study of this size is far from conclusive, the results tell an interesting story. Of the embedding models examined, Google's Universal Sentence Encoder (lite) and Facebook's 's InferSent model clearly have the most effective geometries for this task. Both models were able to index 75% of the free-form text descriptions correctly, and even when they selected an incorrect label, they usually selected it from the right general category of answers. We find this impressive and evocative of future possibilities. The question quickly arises: If models that were trained without this specific indexing task with reasonable accuracy, what might model
application in mind can accomplish accomplish if were trained with the specific objective of facilitating knowledge retrieval via vector offset methods? Further thoughts in this direction are outlined in Section
4

---

Under review as a conference paper at ICLR 2020
Skip-Thought
USE lite
BERT
GPT-2
InferSent
Trunsf-LL XLL
Donald Trump
actor
politicion
politician
pay
politician
actor
Taylor Swift
actor
polirician habitat
jack
actor
cocker spanicl
politician
actor
olympic
dog
animals
parakeet
indigenous
birds newspaper
actress
outcome
bird newspaper newspaper city
The Daily Planet
newspaper
new newapaper
newspaper newspaper (xd7
newspaper
New York Tribune
newspaper
newspaper
newspaper
newspaper
Topeka
city
country
city city auckland city
charter
Vienna
city
city
egyptian
city zealand city
city
New Zealand
politician
country city
newspoper
country
Ivory Coast
actor
curopean
city
Figure 6: Relation extraction. Given starting node label, vector offsets were used to identify relations that fall loosely under the 'is-a' umbrella, such as "occupation', "instance-of" and subclass- of". Relations that could be verified in WikiData bolded. Italicized orange text indicates relations that are not reflected in WikiData, but that perhaps should be.
5 EXTRAPOLATION: DISCOVERING RELATIONS
In this section we demonstrate that, given only few exemplars of relation, linguistic embedding mod- els are able to generalize that relation to new start- ing nodes. Figure[]shoor amples used for this task: Three public figures, one
label
relation
Tom Cruise
actor
the canonical relation ex-
Bill Gates
entrepreneur
Hillary Clinton
politician newspaper
newspaper. one city, one type of animal, and one country name, each of which is paired with an ob- ject that fulfills the relation A, is-a Bj. We use this data to extrapolate new relations for ten previously unseen starting nodes in the following manner.
The Daily Herald
Paris
city mammal
kitten
The United States of America
country
Figure 5: Example texts A, and B; used to define the vector V between a description and the corresponding object
Given an input node label z, set of possible output nodes (4,,... (, and set of example pairs (A,, B,), where each A, is a node label and each B, is the label of an object that fulfills the desired relation, extrapolations to new input nodes can be calculated in the following manner:
1. Create a set of canonical vectors V, B, - -A,
2. For each previously unseen node label æ convert x into a vector representation v V Veearest, where Vnearest is the canonical vector whose
3. Find the indexing point p initial point A, has the closest cosign distance to
4. Search the set of possible output nodes to find I* argminni disteos(li.p))
Note that, when indexing as per Section 4.2 we calculated a single canonical vector V based on all of the exemplar (A, B2) pairs. When finding relations, however, we seek to match the given exemplar relations as closely as possible. Thus we use suite of vectors V,, selecting the one whose source word A, is most similar to the input node vector v. This relational vector Vi then used to query the knowledge base.
Case study results are shown in Figure σ The ten input node labels we used were chosen before any of the results had been calculated, and were not changed afterward. The output label set L consisted of the 5000 most common words in the FastText (Bojanowski et al. (017)) token set.
For the purposes of this study, we defined a valid result al" label that fulfils a WikiData (Vrandečid Krötzsch| 2014) relation of the type 'instance-of", 'subclass of", or 'occupation' (all of which can be considered variant of the 'is-a' relation exemplified in Figure 5) with respect to the input node. The answer 'newspaper' was accepted as being equivalent to the WikiData node 'fictional newspaper', which was not present in our label set L.
Five of the six embedding models produced valid 1' on at least 5 or our 10 case study words. The InferSent and Transformer-XI models seemed particularly adept representing knowledge in this way. Particularly interesting are the labels shown orange italicized text. These are terms that do not appear in the WikiDDta results for each source node, but which arguably correct anyway.
5

---

Under review as a conference paper at ICLR 2020
InferSent; politician
InferSenta
InferSents
InferSeuta
Donald Trump
businessman
citizen
politicians
Taylor Swift
actor dog bird species
actress
actors
singer
cocker spaniel
dogs
animal
wildlife
parakeet
animal
birds
Daily Planet
newspaper newspupers newspaper
magazine
web
The New York Tribune
newspapers
magazine
web
Topeka
city
town
county
township
Vienna
city
town
borough
manicipality
New Zealand
zealand
city
township coast
county
Ivory Coast
city
coastal
municipality
Figure 7: Relation extraction using the InferSent embedding model, showing the first four query re- sults. Bolded text indicates relations that exist in WikiData. Orange italicized text indicates relations that are not present in WikiData, but that are arguably correct. Green italicized text indicates plural versions of valid relations found in WikiData.
This demonstrates the powerful potential of linguistic embedding spaces to extract relations which are lacking from more traditional knowledge representations.
To further the case study, we took closer look the top four label results given by the InferSent embedding model for each of our source nodes (see Figure 77. Source nodes are listed on the left; labels are listed left to right in order of increasing distance from the indexing point p. Examination of the extracted labels reveals innovative and potentially useful information that would not necessarily be encoded in a traditional hand-curated know ledge system: For example, that the President of the United States is also citizen, or that cocker spaniel is an animal as well as dog. Taylor Swift is correctly identified as both an actor and a singer, while the results for Vienna include the insightful result that it is municipality as well as city.
LANGUAGE IN THE WILD: MAPPING HIGH-LEVEL COMMANDS TO ACTION PRIMITIVES
One way to test the amount of common-sene sense knowledge intrinsically present in language repre- sentation is to see whether it can facilitate natural language reasoning: For example, given set of action primitives A = {a], On J. set of action labels T., and a mapping M(..) - A, the question arises: Can the geometric structure of the embedding space be used to map incoming commands from a human user to the correct action primitives?
To explore this question, we created set of 58 user directives and 9 action primitive labels for hypothetical scenario in which human controller is giving voice commands to an unmanned aerial vehicle (UAV). Based the user directive the UAV must determine which behavior to execute. This is done in the following manner:
1. For each human utterance u received:
(a) Convert the u into vector representation v
(b) Find the best-matching action label I' argminge distcos (1,1)
(c) Execute the motor primitive M(1
Results are shown Figure g We begin by observing that none of the embedding spaces is par- ticularly good at this task, however, all of them are able to outperform random baseline. This indicates that significant amount of common-sense knowledge implicitly encoded within the sentence representations. The key question is: How do create embedding spaces in which this common-sense knowledge is more effectively structured?
We begin by noting that the InferSent model, which was trained using recurrent neural architecture that was optimized for only a single training task, has markedly better performance than models which trained many tasks simultaneously. The second second-highes, highest performance was attained by the Skip-Thouge Thought architecture, which was also trained exclusively on a single task (Kiros et al. 2015). This suggests that the current trend of training powerful architectures as many tasks possible,

---

Under review as a conference paper at ICLR 2020
human directive
action label
'advance three feet' 'fowward, please'
go forward
80 forward
fastert
'increase altitude"
go higher
"try to touch the ceiling'
go higher
'hold still'
stop
deiz pooned
mmaintain position"
stop
Figure 8: Left: Example directives and action labels used in the UAV control task. Right: Illus- tration of the UAV control task. Sentence represenations are used map user directives action labels, which in turn can be mapped to direct motor controls.
Skip Though
34.48%
USE lite
29.31%
BERT
22.41%
GPT-2
15.52%
InferSent 48.28%
Transf-XL 31.03%
random
11.11%
Figure 9: Classification accuracy a UAV control task in which human utterances mapped to the most similar action label based on cosign distances within the embedding space.
while undeniably effective at improving performance at transfer learning (Devlin et al., 2018 Cer al., 2018b), may not be the best way produce sentence representations with strong internal semantic structures.
EVALUATION METRICS FOR REPRESENTATIONAL KNOWLEDGE BASES
When evaluating linguistic embedding models, researchers often evaluate model performance based on cross-task generalization. In this paradigm, the learned representations are used as input features for a small, often single-layer, network that leverages pre-trained structure. Thus the quality of the embedding model is implicitly defined in terms its facilitation of transfer learning, property which appears strongly correlated with the use of multiple simultancous training tasks.
There is of course, nothing inherently wrong with this approach. But if wish train embedding models for specific and targeted use as a common-sense knowledge repository, then transfer learn- ing becomes less relevant. Instead, choose to focus on the semantic geometries of the learned embedding space.
In this section, propose three properties that seem correlated with high performance the studies this paper. We observe that these desired properties can either be induced implicitly, by selecting learning tasks which only be mastered when the desired embedding properties present, or explicitly, by directly incorporating a measurement of the desired property within the loss function of the neural network.
1. Analogical coherence. The analogical properties observed in word2vec, GLoVE, FastText, and other single-word embedding spaces should be preserved in the trained sentence-level are preserved.
vector space such that offset relationships like :adrdd: At the phrase or sentence level, this should extend to relationships like 'if you drop :'ll':'it will bounce':: 'i' you drop glass': will break'.
2. Semantic alignment. Single-word embeddings should be located closer to sentences that contain those words than to conceptually equivalent sentences that do not contain them. Sentences that express similar ideas should be located near another despite variations in syntax or structural complexity, and arbitrarily "bags words" should be located close to sentences in which those words appear.

---

Under review as a conference paper at ICLR 2020
negation detection
clause relafedness
Google analogy
Skip-thought
61.48%
20.53%
50.86%
USE lite
77.78%
2.48%
52. 52.12%
BERT GPT-2 InferSent Transf-XL
89.48%
6.02%
46.56%
61.19%
38.23%
6,47%
97.48%
48.50 %
81.81
26.37%
17.88%
47.08%
Figure 10: Classification accuracy on three tasks related semantic geometry. The highest value in each column is bolded.
3. Polarity displacement. A sentence and its negation should be located far from each other along at least one basis dimension of the space, and the negated sentence should be located close to non-negations that nevertheless convey the same concept. (For example, 'The room is not empty" should be located farther from the statement 'The room is empty" than it is from sentences such as "The room is occupied" or 'The room is full'.)
The evaluation framework introduced by zhu et al. (Zhu et al. 2018) includes metrics that are relevant to of these properties. Semantic alignment measured by clause relatedness metric that requires the distance between a sentence and one of its embedded clauses to be less than the distance between the original sentence and its not-negation, while polarity displacement is measured by the negation detection task. The third property, analogical coherence, can be effectively measured using the Google Analogy Test Set introduced by Mikolov et al. (Mikolov et al. 2013a).
Figure 10 shows the performance of four waae-of-the-an embedding models on these tasks. The InferSent model outperforms the other models at all three tasks, just showed the highest perfor- mance on our (admittedly small) indexing, relation-extraction and UAV control case studies. Even more interestingly. the only other model to score greater than 20% at all three tasks was also the pecond-highess scoring algorithm on the UAV control task and tied for second on the relation extraction task (although it failed miserably on the indexing case study).
Although more extensive studies remain to be done, it seems to be the case that high performance on the evaluation tasks in [0 is correlated with strong performance in at least some applications of the embedding model common-sense knowledge base.
CONCLUSION
As the search for more efficient and effective embedding spaces progresses, it is imperative that researchers think critically, not just about the performance of the embedding space on established benchmarks, but on the potential of linguistic embedding spaces for charting new territory.
In this paper, we have examined some of the capacities (and weaknesses) of current sentence-level embedding spaces, and have proposed set of criteria that may help guide the development of future embedding spaces. In particular, we have re-examined basic industry assumptions about how these embedding spaces are created, how they can be manipulated, and for which applications they are best suited. If we wish to extend analogical reasoning beyond elementary word vector calculations, need new kind of embedding space, one that functions not merely as useful input features for downstream tasks, but rather as valuable common-sense repository in its own right: A knowledge base that can be queried mathematically solve real-world problems with little or no direct training. In order to realize this potential, however, new embedding models must be developed which have been optimized for such applications.
We have proposed three broad training criteria Analogical coherence, Semantic alignment, and Polarity displacement- which show signs of correlation with performance on knowledge extraction tasks, and have identified evaluation datasets within the existing literature that measure the extent to which learned sentence representations satisfy these criteria. We hope that, in addition to continued efforts to optimize linguistic embedding models based transfer learning, researchers in the field will begin to explore the training and application of embedding models as common-sens knowlede repositories.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5:135-146,
2017.
Tolga Bolukbasi. Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. Man to computer programmer as woman is to homemaker? debiasing word embed- dings. In D. D. Lee, M. Sugiyama, U. V. Luxburg. I. Guyon, and R. Gamett (eds.), NIPS, pp. 4349-4357. Curran Associates, Inc., 2016. URL http:://appeeee
nips. cttpsperc/pppp//// progr arograamerr ammer- as woman is to homemaker debi asing wimbeedd 6228 man to computer pampptter-poorroee
pdf
Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St. John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, Yun-Hsuan Sung, Brian Strope, and Ray Kurzweil. Universal sentence encoder. CoRR, 10s/1803.11175, 2018a. URL http:
//arxiv.
Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St John, Noah Con- stant, Mario
Steve Yuan, Chris Tar, al. Universal sentence encoder. arXiv preprint arXiv: 1803. 175,, 2018b.
Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, and Antoine Bordes. Supervised learning of universal sentence representations from natural language inference data. CoRR, abs/1705.02364, 2017, http:://///r
abs 1705. 02364
Alexis Conneau, German Kruszewski, Guillaume Lample, Loic Barrault, and Marco Baroni. What you can cram into single vector: Probing sentence embeddings for linguistic properties. arXiv preprint arXiv: 01070, 2018.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04880, 2018.
Nancy Fulda, Daniel Ricks, Ben Murdoch, and David Wingate. What can you do with a rock? affordance extraction via word embeddings. In Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI-17, pp. 1039-1045 1045, 201 7a. doi: 10.24963/ijcai. 2017/141141
2017/144. URL/tttps:///ct:
Nancy Fulda, Nathan Tibbetts, Zachary Brown, and David Wingate. Harvesting common-sense navigational knowledge for robotics from uncurated text corpora. In Proceedings of the First Conference on Robot Learning (CoRL), 2017b.
Ryan Kiros, Yukun Zhu, Ruslan Salakhutdinov, Richard S. Zemel, Antonio Torralba, Raquel Urta- sun, and Sanja Fidler. Skip-thought vectors. pp. 3294 -302, 2015.
Quoe Le and Tomas Mikolov. Distributed representations of sentences and documents. In Pro- ceedings of the 31 International Conference on Machine Learning (ICML-14), pp. 1188-1196, 2014a.
Quoe V. Le and Tomas Mikolov. Distributed representations of sentences and documents. CoRR, abs/1405 1401.4053..33 2014b.
H. Liu and P. Singh. Conceptnet - a practical commonsense reasoning tool-kit. BT Technology Journal, 22(4):211-22) 221, 2004. ISSN 1997-1995... doi: 000002208BBTT110000077004433333 URL httt:://ds www.dooo.....oooooo 10t.:/10.100///00 B:TTTJ. 000.0000000044000 454214
Yang Liu, Zhiyuan Liu, Tat-seng Chua, and Maosong Sun. Topical word embeddings. In In Twen-
ryNinth AAAI Conference on Artificial Intelligence, 2015.
Cynthia Matuszek, John Cabral, Michael Witbrock, and John Deoliveira. An introduction to the syntax and content of cyc. In Proceedings of the 2006 AAAI Spring Symposium Formalizing and Compiling Background Knowledge and Its Applications to Knowledge Representation and Question Answering, pp. 44 49, 2006.

---

Under review as a conference paper at ICLR 2020
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word represen- tations in vector space. CoRR, 1bs/1101,3781, 2013a.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representa- tions of words and phrases and their compositionality. In C. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger (eds.), NIPS, pp. 3111-3119. Curran Associates, Inc., 2013b.
Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. Linguistic regularities in continuous space word representations. Association for Computational Linguistics, May 2013c.
George A. Miller. Wordnet: A lexical database for english. Commun. ACM, 38(1))339 Novem- ber 1995. ISSN 0001-0782. doi: 10.1 10.1145219777..11944. URL
htt.://di.cccccoooommmommooo. .cm.
10. 10.1145/219717 2t1.211.7799994.... 219748
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners.
Ashutosh Saxena, Ashesh Jain, Ozan Sener, Aditya Jami, Dipendra Kumar Misra, and Hema Swetha Koppula. Robobrain: sarge-scale knowledge engine for robots. CoRR, abs/1412.0691 2014.
URL http: org abtg//bss/1422 0691
Peter D. Tumey and Patrick Pantel. From frequency meaning: Vector space models of semantics.
J. Artif. Int. Res,, 37(1):141-188, 2010. ISSN 1076-9757.
Denny Vrandecic and Markus Krotzsch. Wikidata: A free collaborative knowledgebase. Commun.
ACM, 57(10):78 85, September 2014. ISSN 0001-0782.
doi: 10.1145/262948,, URL http:
7/doi. acm. 0rg/10.1145/262348
Denny Vrandečić and Markus Krötzsch. Wikidata: free collaborative knowledge base. 2014.
Xunjie Zhu, Tingfeng Li, and Gerard De Melo. Exploring semantic properties of sentence embed- dings. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguis- fics (Volume 2: Short Papers), pp. 637,637, 2018
10


Reviews:
[{"review_id": "rkljjJJ0tS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper provides an overview of methods to use embeddings for texts as common-sense knowledge. It mentions many aspects where embeddings can be used as common-sense knowledge. However, the paper lacks both novelty and in-depth analysis.  Most methods proposed are basically computing the cosine distances between language embeddings pairs and find the closest one. It's hard to imagine how to scale up this process in real applications. And most of the analyses are based on cherry-picked examples rather than evaluation with quantitative metrics. ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "HkeJHnHy5r", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "- This paper claims to provide a “alternate” view of pretrained embedding as commonsense repository. But this is not a new view at all. It is well known embedding can encode commonsense knowledge, which is a reason it helps a wide variety of recent commonsense related tasks (and there has been considerable analysis on what kind of coomonsense is learned and helpful).\n\n- The paper also claims to propose three training criteria, which are not new as well. In fact, much work has been performed to learn representation for different semantic orientation, dimensions and relations (e.g., polarity like full/empty is one special case of contrasting meaning, which has been studied a lot in the distributed representation paradigm for years).\n\n- The paper has not been carefully written yet; e.g., even in the abstract, there is a typo like “typically trained an evaluated”. \n\nI do not think this paper, claimed as a position paper, adds any new positions to the existing literature. I do not recommend it for the conference.\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BygwytnZ5H", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This is a position paper - it discusses overall about multi-word embedding (sentence) level and how it can be leveraged in multiple applications. Also, it discusses some open opportunities on how the embeddings can be used as a base knowledge source and some challenges in them\n\nPros:\n1. The examples and analysis of embeddings involved in the paper is detailed\n2. The authors have run lots of experiments to position their idea correctly.\n\nCons:\n1. I find many of their intuitions well established in the literature - the intuition of geometric structures in the embedding space, the intuition of incidence angle. The ideas from section 4,5,6 seem to be rather obvious extension of what we know\n\n2. Especially, in Section 4,5 the idea of finding indexes, is nothing but the projection of x on the mean of (A,B) - which is more of a common practice. I find both these ideas of finding indexes and extrapolation, pretty much recurrent and existing in literature.\n\n3. What I would be really interested is, a new training algorithm for learning embeddings, that would be able to answer all linear algebra based questions in the embedding space. Rather, this paper analyses the pros and cons of 6 existing models. In my personal opinion, this is half work done for a very good paper.\n\n4. Personally, I am not very appreciative of the use of  \"Common Sense\" in this paper. There is a set of papers in the literature (atleast in vision domain), where they actually try to learn the common sense from the underlying data. The use of common sense in this paper does not align with that usage and hence, is misleading for me.\n\n5. From reading a position paper, I would ideally want to see ideas worth of atleast 2-3 PhD thesis. I do not find such compelling ideas in this paper.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents an analysis of the kind of knowledge captured by pre-trained word embeddings. The authors show various kinds of properties like relation between entities and their description, mapping high-level commands to discrete commands etc. The problem with the paper is that almost all of the properties shown in this work has already been established in existing literature. In fact, the methods presented here are the baseline algorithms to the identification of different properties presented in the paper.

The term common-sense which is used often in the paper is mischaracterized. In NLP literature, common-sense is something that is implicitly understood by humans but which is not really captured by language. For example, going to a movie means you need parking is something that is well-understood by humans but is not implied by the language of going to the movie. The phenomenon described by the authors is general language processing.

Towards the end the evaluation criteria for embedding proposed is also a well-established concept, its just that these metrics are not part of the training mechanism as yet. So if the contribution was on showing how those metrics can be integrated in training the embeddings, that would be a great contribution.

I agree with the reviewer's critics and recommend a rejection as of now.
Decision: Reject