Measuring Numerical Common Sense: Is A Word Embedding Approach Effective?
Authors: Hiroaki Yamane
Chin-Yew Lin
Tatsuya Harada
Venue:  (2020)
Keywords: numerical common sense
word embedding
semantic representation

Abstract: Numerical common sense (e.g., ``a person with a height of 2m is very tall'') is essential when deploying artificial intelligence (AI) systems in society. To predict ranges of small and large values for a given target noun and unit, 
previous studies have implemented a rule-based method that processed numeric values appearing in a natural language by using template matching. To obtain numerical knowledge, crawled textual data from web pages are frequently used as the input in the above method. Although this is an important task, few studies have addressed the availability of numerical common sense extracted from corresponding textual information. To this end, we first used a crowdsourcing service to obtain sufficient data for a subjective agreement on numerical common sense. Second, to examine whether common sense is attributed to current word embedding, we examined the performance of a regressor trained on the obtained data. In comparison with humans, the performance of an automatic relevance determination regression model was good, particularly when the unit was yen (a maximum correlation coefficient of 0.57). Although all the regression approach with word embedding does not predict values with high correlation coefficients, this word-embedding method could potentially contribute to construct numerical common sense for AI deployment.

Under review as a conference paper at ICLR 2020
MEASURING NUMERICAL COMMON SENSE: IS A WORD EMBEDDING APPROACH EFFECTIVE?
Anonymous authors Paper under double-blind review
ABSTRACT
Numerical common sense (e.g., person with height of 2m is very tall") is essential when deploying artificial intelligence (Al) systems in society. To predict ranges of small and large values for a given target noun and unit, previous studies have implemented rule-based method that processed numeric values appearing in a natural language by using template matching. To obtain numerical knowl- edge, crawled textual data from web pages are frequently used as the input in the above method. Although this is an important task, few studies have addressed the availability of numerical common sense extracted from corresponding textual in- formation. To this end, we first used a crowdsourcing service to obtain sufficient data for subjective agreement numerical common sense. Second, to examine whether common sense is attributed to current word embedding, we examined the performance of regressor trained on the obtained data. In comparison with hu- mans, the performance of an automatic relevance determination regression model was good, particularly when the unit was yen (a maximum correlation coefficient of 0.57). Although all the regression approach with word embedding does not pre- dict values with high correlation coefficients, this word-embedding method could potentially contribute to construct numerical common sense for Al deployment.
INTRODUCTION
We live in a world where common sense is basic requirement: if person's weight is under 50 kg, then they are likely to be very thin; if a person's weight exceeds 80 kg. then they are likely to be hefty. A similar concept
be extended to situations involving money. If new high-spee laptop is $300, then it is considered to be quite cheap. Therefore, it is essential to account for common sense future artificial intelligence (AI) systems.
In a society that robots communicate with humans, these robots have to estimate how large the thing is (e.g.. If robot mistakenly estimate the object weight, it may harm humans and/or robots themselves).
There are quantitative units such physical scales (meter, kilogram, etc.) and money; studies have treated these scales as a unit.
Distributed semantic representations have gained much attention and showed a great performance in varieties of NLP tasks. Mikolov et al. (2013) developed skip-gram model to obtain semantic representations by using a novel word-embedding approach. Their distributional word embedding exhibited a remarkable performance in analogy tasks (e.g., "King Man + Woman = Queen").
In more recent studies, Gladkova et al. (2016) prepared varieties of analogy pairs named the Bigger Analogy Test Set (BATS) and compared GloVe (Pennington et al. (2014)) and count-based (i.e., PPMI + SVD) model (Church Hanks (1990). our point of view, we assume word themselves have numerical information. That is, similar words share similar units and thus concrete words contains numerical information. The primary motivating assumption is that, similar to analogy, con- textual flow in word sequences would have specific repetitive representations, and thus assume that this specific word representation (c.g., co-occuring unit word such as gram) would include the numerical knowledge for given target object.

---

Under review as a conference paper at ICLR 2020
Here, we attempt to analyze the relationship between human-generated and extracted common sense by using word embedding. We assume that common sense is attributed to distributional word em- beddings such that a trained regressor can predict numerical values.
The uniqueness in this field lies treating abstract concepts as common sense. Other than the size of the object, we focus on nonphysical scales, such as temperature and weight, well subjective scales, such as money. Moreover, a challenging problem when estimating numerical common sense that there are few datasets with labeled numerical values. This issue is tackled in our study through a word embedding approach. The contributions we make are two-fold:
We used a crowdsourcing service to obtain a sufficient number of subjective agreements on numerical common-sense data.
We examined the predictabilities of regressor trained using word embedding and crowd- sourced data. We also examined whether a concatenated semantic vector can effectively predict numerical values.
The remainder of the paper is organized as follows: After introducing related works in Section 2 we describe the collected datasets in Section 3 The results of the experiments conducted using the obtained data are then presented in Section Subsequently, in Section [5] the obtained results discussed and interpreted. The study is summarized and future scope is described in Section
2 RELATED WORK
2.1 APPLICATIONS THAT TREAT NUMERICAL VALUES
Narisawa al. (2013) proposed methods that use large amount of text, related to the abovemen- tioned units, to determine common sense text patterns. Their approach estimates the range of the units co-occurring within a context and examines whether a given value large, small, or normal based on its distribution. Their approach utilizes textual patterns with which speakers explicitly express their judgment regarding the value of numerical expression. Similarly, (Hayakawa & Hagi- wara 2014) used snippets from web search engine to extract more scalable but lesser amount of text data, related to the units, through the design of heuristic functions that enable the system to han- dle input frequencies, which often very noisy. These two approaches require careful intuitions when determining the thresholds of a proposed method or building an effective function for each numerical value. Qin et al. 2018) further developed a numerical variable extraction mechanism. They modeled the joint probability of data fields, texts, phrasal spans, and latent annotations with an adapted semi-hidden Markov model and imposed a soft statistical constraint to further improve the performance of the model.
Another approach focuses using different modalities. A 165 cm human cannot be taller than a I km long bridge. This concept has a wider range of applications. Bagherinezhad et al. (2016) considered object sizes using computer vision technique.
In the context of only text, particularly in relation to numerical common-sense extraction, most methods rely large amount of data with carefully designed heuristies. As such, it would be beneficial if it was possible to extract common sense only using certain human evaluations.
Forbes al. Forbes & Choi (2017) present an approach infer relative physical knowledge of actions and objects along five dimensions (c.g., size, weight, and strength) from unstructured natural language text. They reported that it is possible to extract knowledge of actions and objects from language and that joint inference over different types of knowledge improves performance. Yan et al. YYang et al. (2018) proposed and assessed methods for extracting one type of commonsense knowledge, object-propert comparisons, from pre-trained embeddings. They also showed that an active learning approach that synthesizes common-senss queries can boost accuracy. Elazar et al. Elazar et al. (2019) propose unsupervised method for collecting quantitative information from large amounts of web data, and use it to create a new, large resource consisting of distributions over physical quantities associated with objects, adjectives, and verbs.
2

---

Under review as a conference paper at ICLR 2020
NCS-50X1
object
unit number
NCS-60x3
object
unit
family member milk winter climate
laptop
yen
apple coffee cup
centimeter
degree Celsius
meter kilogram
university entrance rate annual income
percentage
piano
yen
private jet plane
gram
wine
Table 1: Examples of objects and units for each dataset. Note that the object and unit need not
correspond.
2.2 STUDIES ON DISTRIBUTED SEMANTIC REPRESENTATION
The approach proposed herein is closely related existing semantic vector models (i.e., word em- bedding). More recent studies used global contexts Pennington et al. (2014) or incorporated other external data, such brain activity from fMRI data Fyshe al. (2014), improve performance. fastText [Bojanowski et al. (2017) improves performance by considering subwords in the model. Jeawak al. (20T9) imcoporated geographical information into semantic representations deal with locations for flicker images.
2.3 THEORETICAL BACKGROUND ON NUMERICAL WORD EMBEDDING
We describe why numerical values are related to word embedding. In the context of distributed word embeddings, "count-base" and "prediction-basedd" method has something in common. As a traditional count-based method, hefore conducting SVD, matrices of Positive Pointwise Mutual Information (PPMI) is used obtain word embeddings (Church & Hanks (1990)) More concretely,
P(r,y)
PPMI m0.r(0 PMT(x, u)) max 0, 0, P(x), Ply))
where P(z) and P(y) are frequencies of each word and P(x. y) indicates co-occurrence of the two words. As such, word that co-occur with words in group and another word that co-occur with words in the smilar group will have similar embedding. Numerical representation will on this point will be obtained in this case.
Similarly, for current prediction- word embeddings, loss fucntion of continuous bag-of- wordds-words (CBOW) is represented as:
1 LCBOW T
Σ log ((wx)w)-1,;(++1)
where W4-1, W1+11 predict Wz, resulting in surrounding words will have similar embedding. In addition, loss function of Skip-gram fhip-gram is calculated as:
T
1
Lskip-gram
log (ogPP(w-llw)) + Jogp(Ww+1leee
(3)
T
where We is to predict w;-1, 11++1 Because Skip-gram has to handle one word to predict multiple words, we assume Skip-gram will obtain more information about numerical values.
DATA COLLECTION
We used the Japanese crowdsourcing service Lancers collect data on numerical common sense Table Iists examples of objects and units for each dataset.
We asked participants to annotate these objects and units indicate "from which value you think it's large" and "from which value you think it's small." Note that for temperature, this "large quantity"
We plan to make the dataset available the public after publication

---

Under review as a conference paper at ICLR 2020
03-
63-
cont
cool.
Corl.
co0l.
cont
cor
corl
Codl
COMP
cod
cont
corT.
cor.
carl
corl
wwwtarntion Imn.
seperrtbeiiggggeg (ogred)
shinp lon)
dialorany fom))
ditts
day BeEHAII (mgghe))
sleeping (oarg)
gooo chochu
widday
bo5y
Figure 1: Some examples from NCS-50xI. The histogram indicates the items' averages and the bars
represent the standard deviations.
will be "high temperature." For Numerical Common Sense including 50 items with one pair (NCS- 50x1), we asked 10 participants create 50 pairs of samples with different units. For Numerical Common Sense including 60 items with three pairs (NCS-60X3), we asked 10 different participants construct 60 pairs of samples with three different units. In total, we collected 50 60 x = 230 pairs of units and values. We also collected participants' confidence (5-rated evaluation: is least confident and is most confident) for their answers.
3.1 CONSTRUCTION OF NCS-50XI
In 1,S-50x1, we selected terms that represent an "object" and "unit" based on the results obtained by Hayakawa & Hagiwara (2014) and included all their 26 examples. Moreover, we added other 24 object and unit pairs. This dataset was collected on February 28, 2019.
Figure illustrates some examples that were collected. The ranges of "small" and "large" differ depending on the term. For example, large in train transition is around 10 times more (with large variance) than small, while both large and small in body temperature have smaller variances. More- over, there are terms whose confidence from participants more than others. Compared with guess- ing large/small of temperatures in hot spring, participants are confident when predicting large/small of sleeping hours.
3.2 CONSTRUCTION OF NCS-60X3
In NCS-60x3, for a more exhaustive study on variation concept, we limit objects and units using WordNet Miller et al. (1990) and brain-based knowledge representation Naselaris et al. (2009) as a reference. In the supplementary material provided by Naselaris et al. (2009). a handful of word concepts to check the brain decode and encode predictabilities were shown; assumed these concepts could be applied to crowdsourcing on reasonable seale. We use 60 objects considering the above concepts and the obtained crowdsourced data. Particularly, animate, artificial, and food objects were taken into account. This dataset was collected on March 3, 2019.
We depict the histogram of 18 example items in Figure 2 and
As the figure shown, the results from participants are very diverged. To make a fair comparison, we use the same unit through size, and weight (i.e., is converted to em by computing "x 100").
4

---

Under review as a conference paper at ICLR 2020
196
2.000
18:00
200000
6.000
2.00
0/
FDE bedd
XBX
19003
18.00
19:00
3803
X.01
140
2003
2.50
Figure 2: Histogram of money-small 18 out of 60 items. The X-axis is Yen and the Y-axis represents number of participants.
Figure 3: Histogram of sizll 18 of 60 items. The X-axis is centimeter and the Y-axis represents number of participants.
5

---

Under review as a conference paper at ICLR 2020
Expl: SinglelConeatenatee Word Voctor
Exp2: Concatenated Word Vector Vs. Human
Word Embeddings
Train Phase
+ Label (small/large values)
Prediet Labels
Test Phase
(small/lange values)
Figure 4: Diagram of experimental procedure
EXPERIMENT FOR NUMERICAL VALUE PREDICTION
We analyze regressors trained on the word embeddings along with human evaluated data examine whether common sense is attributed to current word embedding. Figure S shows a diagram of this experiment. After checking performance on various units with/without word embedding concatena- tion on different regressor in Experiment I, we compare the regression performance on each word embedding type trained on different corpus. Then we also compare that performance with humans in Experiment 2. We intend to 1. investigate overall tendencies using NCS-50x1 in the first ex- periment because we include terms (objects and units) from one of the pioneer works of numerical common sense (Hayakawa & Hagiwara (2014)), and 2. obtain more detailed points including results on selected units and types of word embeddings comparing with human participants in the second experiment.
Section 3.1 provides more details on the experimental settings and Section 3.2 describes experiments 1 and 2, where we use NCS-50x1 and NCS-60x3, respectively.
4.1 EXPERIMENTAL SETTINGS
We describe the details about word embeddings and machine learning algorithms that we use. We followed the procedure described below to prepare for the experiment.
We use word embeddings trained on Japanese Wikipedia) which was downloaded in Feb 2015. To simplify the problem, when there were multiple words in an object, feature vectors were averaged. Note that when tokenizing, we used MeCab (Kudo et al. (2004)). We then obtained word embeddings based on a Continuous Bag-of- Words (CBOW) model with 200 dimensions. Henceforth, we refer to this as "w2v-wiki."
We use fastText word embeddings Bojanowski et al. (2017) trained Japanese Wikipedia that are available at Grave et al. (2018). The model was trained using CBOW with position- weights, in dimension 300, and character n-grams of length 5, window of size 5 and 10 negatives. Henceforth, we refer to this setting as "fT-wiki."
We use word embeddings trained on crawled Japanese web texts Yata), Nihongo Web Corpus (NWC). The texts composed of approximately 100 million html documents (3.25TB in total), which resulted in 396 GB of plain texts. Note that when tokenizing, we used MeCab with the NEologd dictionary. The model was trained using CBOW, in dimension 500, with a window of size 15. We refer to this setting as "w2v-nwe."
For experiment 1, we prepared different types of word vectors; object only, unit only, and concatenated feature, respectively. For object only (e.g. "suimin": sleeping) and unit only (e.g.. "jikan": hours), word embeddings are directly extracted from "w2y-wiki" and re- gressors are trained and tested. For concatenated features, those two word vectors (namely object and unit) were concatenated, which resulted in 400 dimensions for the concatenated features. For Experiment 2 where we use NCS-60x3 dataset, we also use "fT-wiki" whose concatenated embedding dimension is 600, and "w2y-nwe" whose one is 1000.
We employed a linear regressor with the least square (LS) eттоr penalty and a linear re- gressor with an automatic relevance determination (ARD) prior. We also use random forest

---

Under review as a conference paper at ICLR 2020
object only
Mean Abs. Err.
Corr. Coef.
ARD
1.20 x 10"
0.080
LS
1.15 10?
0.011
RF
1.04 10"
0.002
unit only
ARD
1.20 10'
N/A
LS
1.10 10º
N/A
RF
9.91 10
N/A
Concat. feat.
ARD
1.20 x 10'
0.254
LS
1.30 10?
-0.035
RF
9.08 x 10
0.181
Table 2: Results obtained from linear and random forest regressors. Note that Corr. Coef. indicates Pearson's 's correlation coefficient and MAE is mean absolute error
(RF) regressor for comparison. In the RF, the number of decision trees was set to 10; in ARD, the parameters were set to alphal=le-06, alpha2=1e-06, lambdal=1e-06, and lambda2=1e-06. The LR input will be normalized before regression by subtracting the mean and dividing by the 12-norm. After average numerical values were obtained from 10 subjects, conducted 10-fold cross validation. To examine performances of those regressors, we use the mean absolute error (MAE) and the Pearson's 's correlation coefficient.
To compare the predictability between CBOW and Skip-gram, and increament of word embedding dimensions, we use Japanese Wikipedia and change the dimensions from 100 to 500 in experiment2.
We used gensim Rehurek Sojka (2010) for training word embeddings and feature extrac- tion, and scikit-learn Pedregosa et al. (201T) for executing machine learning algorithms. We use w2v-wiki for Experiment and all three word embeddings for Experiment 2.
4.2
EXPERIMENTAL RESULTS
4.2.1 EXPERIMENT I ON NCS-50X1
The regressor performance results are shown in Table Essentially, MAE showed high prediction errors and the Pearson's correlation coefficient was quite low. The Concat feat. in ARD appeared to have higher correlation than other non- non-human methods. Therefore, in the next experiment employ word embedding concatenation.
4.2.2 EXPERIMENT 2 NCS-60X3
Table 3 list the results for 60 objects with three different fixed units obtained from linear regressors.
Here, we fixed unit and examined the predictabilities of word vectors trained on human evalua- tion. In Experiment we also compared the results the trained regressor with those of human evaluation.
Humans scores and predictors are calculated in the same manner. Concretely, in 10-fold cross validation manner, correlation between evaluation scores from each subject and averaged other 9 subjects were calculated. Humans performed better than both of regressors, yet ARD had a corre- lation coefficient of 0.57, which appeared to be comparable to that of humans when the unit is Yen and gram.
In most cases, the regressor had better prediction in MAE when the unit centimeter. This may be because there less variation in values. In contrast, the correlation cocfficient of both regressors showed better performance for the unit of gram.
In addition, we examined CBOW and Skip-gram with an increase dimensions. When we tested the data Japanese Wikipedia, observed that the accuracy changed with increase in the dimensions For ADR, the average correlation coefficient across units [Yen, g, cm] was 0.213 with 200-d, 0.186 with 300-d, 0.221 with 400-d, and 0.221 with 500-d. We observed better correlation

---

Under review as a conference paper at ICLR 2020
MAE
Corr. Coef.
unit: YEN
w2v-wiki
IT-wiki
w2v-nwc
w2v-wiki
TT-wiki
w2v-nwe
large ARD
9.07 10"
9.07 10"
9.07 10"
0.360
-0.237
0.571
small on ARD
7.56 x 10
7.56 x 10
7.56 x 10
0.403
0.013
0,418
large LS
2.89 10
3,80 10
1.55 10
0.216
-0.033
0,444
small on LS
2.45 107
3.39 x 10
2.45 107
0.212
-0.118
0.268
large RF
1.38 10
7.56 107
1.49 10
0.127
0.217
0.500
small on RF
x
4.40 10
1.29 107
0.119
0.262
-0.012
large on human
6.08 x 10?
0.753
small on human
4.73 10
0.894
unit: em
large ARD
712
911
695
0.120
0.0070
0.310
small ARD
$1.4
189
142
0.205
-0.018
0.487
large
383
479
0.502
0.111
0,249
small on LS
31.9
284
142
0,270
-0.014
0,460
large RF
339
189
117
0.236
0.259
0.067
small on RF
25.1
105
102
0.148
0.162
0.234
large human
0.828
small on human
51.9
0.905
unit: g
large on ARD
3.99 x 10
3.98 x 10'
3.98 x 10"
0.508
-0,371
0.359
small on ARD
9.21 x 103
9.25 x 106
9.25 10
0.339
-0.265
0.358
large
1.27 10
2.00 10
8.33 107
0.180
0.182
0.118
small on LS
2.08 10a
2.56 107
1.47 x 10
-0.075
-0.266
0.060
large RF
5.52 x 10a
2.60 x 10
1.46 10
-0.047
-0.041
0.041
small on RF large human
1.20 10*
8.61 x 10
1.29 x 10?
0.292
0.092
0.288
3.84 107
0.746
small on human
8.41 x 10
0.774
Table 3: Mean absolute error results for 60 objects with 3 different fixed units obtained from linear and random forest regressors with unit on the left (Yen, cm, and g). Pearson's correlation coefficient results on the right. Bold digits each row indicate higher performance.
8

---

Under review as a conference paper at ICLR 2020
when using Skip-gram for ARD regression. Average correlation coefficient across units was 0.343 with 200-d, 0.378 with 300-d, 0.325 with 400-d, and 0.372 with 500-d.
5 DISCUSSION
We computed the predictabilities of numerical values solely from word embeddings to examine whether they have such information. This idea is based whether corpus has some contextual in- formation of numerical values. Because word embedding models treat context windows, when pre- dictabilities obtained, some kind of numerical knowledge is embedded as co-occurrence. Word embedding is treating co-occurrence and numerical values may be included. In context, objects that are occur is similar and then, text embedding will be possible because the numerical tendencies will be similar. One can imagine that w2v-nwe embedding outperformed in correlation coefficient because of not only its size and diversity, but also contextual window size.
Overall, predicting large was quite diverged on both on regressors and humans.
As indicated by Figure ! numerical values from annotators vary for specific words. As depicted in Figure 2 some metrics especially size had a large variance. For example, the size of some items (e.g., "plastic model" and "audio system") differed more than 1000 times depending on persons when treating large. Basically, it was also difficult to predict money when it comes to MAE. A few unknown but expensive objects such as "private jets" considerably influenced on the result. While our method on MAE did not work well, correlations between predicted numerical values and human evaluations were observed. Prediction by regressor is quite comparable to human in terms of correlation coefficient. As we anticipated, Skip-gram showed a better performance.
CONCLUSION
We are living in world where we need common sense; thus quantifying numerical common sense essential to deploy Al systems in society. We collected and analyzed numerical common-sense values through a crowdsourcing web service. Our contributions are two-fold.
First, we collected data on a subjective yet not fully explored area. We then compared current approaches from a quantitative perspective. To this end, first used crowdsourcing service to obtain sufficient data for a subjective agreement on numerical common sense. We constructed two datasets, namely, NCS-50x 1 and NCS-60x3.
Second, we compared the averaged numerical values from participants on the crowdsourcing plat- form with that of the regressor using word vectors. To the best of knowledge, this is the first approach that tackles this point. In comparison with humans, the performance of an automatic relevance determination regression model was good, particularly when the unit was yen (up to a cor- relation coefficient of 0.57). In addition, predictability increased when we employed the Skip gram. Even though we confirmed positive correlation coefficients, the model was not comparable with the data collected using the crowdsourcing service when it was trained on different units (e.g., when trained on yen and tested on gram).
Although our novel regression approach with word embedding does not always predict values with high correlation coefficients, it exhibited potential to quantify numerical common sense.
In addition, obtaining real numerical distributions would be very helpful in advancing this area. We have a number of large copora with numerical numbers. For example, Wikidata, which was converted from Wikipedia to be precessed by computers can be useful as extention. We believe those datasets can be utilized to expand the size of the numerical common sense dataset. Moreover, it would be more effective to employ more powerful newural regressors.
Simultaneously, we plan to pursue the sophistication of semantic representations of the model.
9

---

Under review as a conference paper at ICLR 2020
REFERENCES
Hessam Bagherinezhad, Hannanch Hajishirzi, Yejin Choi, and Ali Farhadi. Are elephants big- ger than butterflies? reasoning about sizes of objects. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, AAAI' 16, pp. 3449 3456. AAAI Press, 2016. URL
http://dl. acm. wtt/:itatioooonooooo//.. cfm def.midd=3116330
3316389
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5:135- 146. 2017. doi: a0.1162/tacl 00051. URL https:/ www www.accllae aww.wb.www............... www.://wwwww./.aaool..oooooo
017-1010
Kenneth Ward Church and Patrick Hanks. Word association norms, mutual information, and lex- icography. Computational Linguistics, 16(1):22-29, 1990. URL https: //ww:://www... web.
org/ hetg/attoll/ Logy/ 1503
Yanai Elazar, Abhijit Mahabal, Deepak Ramachandran, Tania Bedrax-Weiss, and Dan Roth. How large are lions? inducing distributions over quantitative attributes. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. Italy, July 2019. Association for Computational Linguistics. doi: 10. 10.18553/11/P144 19881188..8881.. URL
Florence,
https: / www. , ac web. org/ antho ogy 9- 1388
Maxwell Forbes and Yejin Choi. Verb physics: Relative physical knowledge of actions and ob- jects. In Proceedings of the 55th Annual Meeting of the Association for Computationnal Lin- guistics (Volume 1: Long Papers), pp. 266-27666666666 Vancouver, Canada. July 2017. Association for Computational I Linguistics. doi: 10.18655//1/P17-00225. URL.tttss: / www. ac web org/ anthol Siechoogy/P17--o
Alona Fyshe, Partha P. Talukdar, Brian Murphy, and Tom M. Mitchell. Interpretable semantic vectors from a joint model of brain- and text- based meaning. In Proceedings of the 52nd An- mual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 489-499, Baltimore, Maryland, June 2014. Association for Computational Linguistics. doi: 1/4115///1/114-1046 URL www. wwwweb..wc......... wttp://nntttooooo//11--1144
Anna Gladkova, Aleksandr Drozd, and Satoshi Matsuoka. Analogy- detection of morpho- logical and semantic relations with word embeddings: What works and what doesn't. In Pro- ceedings of the NAACL-HLT SRW, pp. 47-54, San Diego, California, June 12-17, 2016. 2016.
ACL. doi: 10.18653/1/NN16-2000 URL https: /ww. www.ww.lwwebb...... www.aacleeb..rr///attoo.....
6166/N6620022
Edouard Grave, Piotr Bojanowski, Prakhar Gupta, Armand Joulin, and Tomas Mikolov. Learning word vectors for 157 languages. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC-2018), Miyazaki, Japan, May 2018. European Lan- guages Resources Association (ELRA). URL https:/ wwww.www.oo aclweb. org attpg//thhoog/////
118-1550
Yuta Hayakawa and Masafumi Hagiwara. Quantitative common sense estimation system and its application for membership function generation. Journal of Advanced Computational Intelligence and Intelligent Informatics, 18(5):856-866 2014.
doi: 20.2096/jaaciii221144 p.056..
Shelan S. Jeawak, Christopher B. Jones, and Steven Schockaert. Embedding geographic locations for modelling the natural environment using flickr tags and structured data. In Leif Azzopardi, Benno Stein, Norbert Fuhr, Philipp Mayr, Claudia Hauff, and Djoerd Hiemstra (eds.), Advances in Information Retrieval, pp. 51 66, Cham, 2019. Springer International Publishing. ISBN 978- 3-030- 15712-8.
Taku Kudo, Kaoru Yamamoto, and Yuji Matsumoto. Applying conditional random fields to Japanese morphological analysis. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, pp. 230 237, Barcelona, Spain, July 2004. Association for Computational Linguistics. URL http://wwwww..aa......../ooaooa/
Lancers. Lancers. URL www. www.laances....
10

---

Under review as a conference paper at ICLR 2020
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. Distributed represen- tations of words and phrases and their compositionality. In Proceedings of the 26th Interna- tional Conference
Neural Information Processing Systems Volume NIPS' 13, pp. 3111- 3119, USA, 2013. Curran Associates Inc. URL. http:/ /tbp://dl www.laccmmmmccmmmmmmmmmmmmmmmcc. wotttiiiiiiiiooooooooooio cfm? id=2999792. 2999959
George A. Miller, Richard Beckwith, Christiane Fellbaum, Derek Gross, and Katherine Miller Wordnet: An on-line lexical database. International Journal of Lexicography, 3:235-244, 1990.
Katsuma Narisawa, Yotaro Watanabe, Junta Mizuno, Naoaki Okazaki, and Kentaro Inui. Is 204 cm man tall or small ? acquisition of numerical common sense from the web. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 382-391, Sofia, Bulgaria, August 2013. Association for Computational Linguistics.
URL https: www ac web. www.w..........ooooooooo//ooo htt. ogy/h1lglyppp31331 3-10
Thomas Naselaris, Ryan Prenger, Kendrick N. Kay, Michael Oliver, and Jack L. Gallant. Bayesian reconstruction of natural images from human brain activity. Neuron, 63:902-9115 2009.
NEologd. mecab-ipadii-nneolgs Neologism dictionary for mecab. URL https: /g Chub com/ http::///wmaaapppppc.ccooaoooo/a
RREADD/ ja.ma..
Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake Vanderplas, Alexandre Passos, David Cournapeau, Matthieu Brucher, Matthieu Perrot, and Edouard Duch- esnay. Scikit-leam: Machine learning in python. J. Mach. Learn. Res., ber 2011. ISSN 1532-4435. http:: acm. http://itttiiio....ccccccacccaa. 2078199
Novem-
Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word rep- resentation. In Proceedings of the 2014 Conference on Empirical Methods Natural Language Processing (EMNLP), pp. 1532-1543, Doha, Qatar, October 2014. Association for Computational ogy/
Linguistics. doi: 10.31155v1/D14-111 URL wtt.t:ttp///.m....... www a
web owgp://///www..ooooo./o///o///
Guanghui Qin, Jin-Ge Yao, Xuening Wang, Jinpeng Wang. and Chin-Yew Lin. Learning la- tent semantic annotations for grounding natural language to structured data. Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing, pp. 3761-3771, Brussels, Belgium, -ctober-November 2018. Association for Computational Linguistics. doi: 1.118533/1/D18-1411 URL htttl::/t:://ttt
Radim Rehurek and Petr Sojka. Software framework for topic modelling with large corpora. In IN PROCEEDINGS OF THE LREC 2010 WORKSHOP ON NEW CHALLENGES FOR NLP
FRAMEWORKS, pp. 45-50, 2010.
Wikipedia. Wikipedia japanese dump data, wikimedia downloads. URL https://dumg::
wi wikimedia. otg////wii///.////oooo.
Yiben Yang, Larry Birnbaum, Ji-Ping Wang. and Doug Downey. Extracting commonsense properties from embeddings with limited human guidance. In Proceedings of the 56th An- nual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 644-649, Melboumne, Australia. July 2018. Association for Computational Linguistics. doi: 10.18653/v1/P18-
URL https: /wwwww wwwwww.www........ www.w//wwwwg///aaooooooo////----
Susumu Yata. nwc2010/_
Nihongo web corpus 2010.
URL http://www.-yaaa.jppccoo.
11

---

Under review as a conference paper at ICLR 2020
1.0
Pearson's correlation
0.8
0.6
0.4
0.2
large
small
confidence
0.0
2
4
8
10
Number subjects
Figure 5: Inter-rater agreements of NCS-50x1
A APPENDIX
A.1 DETAILS DATASETS
A.1.1 NCS-50X1
To check the agreement on each participant, we conducted a correlation analysis shown in Figure
5
To calculate the inter-rater agreements, we aggregated the scores from participants, and we com- puted correlation between groups of subjects. In the figure, the X-axis indicates the number of subjects and the Y-axis is the calculated correlation. The bars indicate the standard deviation of the correlation. As the number of the participants increases, reasonably high correlation is observed. Compared with large and confidence, small had higher correlation.
A.1.2 NCS-60X3
The results of a correlation analysis are shown in Oigu the number of participants increases.
In the figure, the X-axis indicates the number of subjects and the Y-axis is the calculated correlation. The bars indicate standard deviation of the correlation. Except for size large, as the number of participants increases, higher correlations were obtained. Because few items (e.g., "plastic model" and "audio system") that are not common for participants were included, seems that estimating sizes, especially large was quite diverged. Compared with other modalities, prices shared similar tendencies, which resulted higher correlation.
Herein, we list results of histograms. Note that these represent values in NCS-60x3 dataset and showing "price-large", "price-small", "size-large", "size-small", "weight-large". "weight-small." and confidence with 5-point scales. Units (which corresponds to the X-axis) are "Yen", "Yen". "cm". "cm", "g", "g", and "1-5" respectively for each page in all figures. Note that the number of the participants is 10 that the maximum number of the Y-axis will be 10 at the maximum. Figure 7,8 g 10, 10 II 12, and [3 show histograms of price-large, pricl-small, size-large, size-small, weight-large, wmilh,- and confidence, respectively.
12

---

Under review as a conference paper at ICLR 2020
1.0
Pearson's correlation
0.8
0.6
0.4
0.2
large small
confidence
0.0
2
4
8
10
Number subjects
Figure 6: Inter-rater agreements of NCS-60x3
A.2 SCATTER PLOT OF EXPERIMENT
We show a scatter plot of test human and prediction Figure 14 fold-cross validation for prediction.
Note that we conducted the 10
13

---

Under review as a conference paper at ICLR 2020
500 800
100000
100030
34000
180000
380000 38.000
50000
7500 79000
100000
150000
3094
1000CE
3000
54040
+000
74.40
150.00
1509
59.0
756
1000
0
20/00
-0004
200000
40008
Gecosit Invertl
Cose
100000
20006
10000
-00000
30008
3000
4.0000
596000
100000
20.00
50.00
7500
30000
2000
1090
Tool Beiterrggggggggo orre
TER000
2000
3:03
10:03
70400
10:300
Laudil drn
21
020
1080
Figure 7: Histograms of price-large

---

Under review as a conference paper at ICLR 2020
TOW
2010
19h4
1500
259
500
150
1004
2000
3804
9100
340
4008
3008
300
3.0
beadphuce
3040
100000
3000
10,00
13.00
100
-04
Aecretd
28.00
40/00
3.69
4009
4066
6.00
9.00
11:00
2.00
10000
2000
40.
1900
60
Scall Brathintty otse
betkdber
101
110
3.01
78000
10100
3.03
19:0
356
7100
100
(dkkig antends
1x
Figure 8: Histograms of price-small
15

---

Under review as a conference paper at ICLR 2020
odbod
206
10:00
25,00
41006
20
1800
64
Sraadh Lownt))
-012
Nodd
Brarheat
2.00
Socilal Lokor
Sexterm
Figure Histograms of size-large

---

Under review as a conference paper at ICLR 2020
rdbed
250
500
20
-ig 2005
7.5
10
Toaal Elen
Hohee bmark)
kettake
Figure 10: Histograms size-small

---

Under review as a conference paper at ICLR 2020
30000
4000
30000
200.89
10000
3,6
38000
5.90
1000
10:00
2000
7000
10.00
1551
37.4
sndo.gra
73000
18.0
1583
30.00
5000
1000
3000
2801
300
250
400
(nvem))
1xT
20/04
10:00
29.00
5004
1906
12000
19690
266
3000
100
1000
1200
gelte 86
Telh Fir www.hhottycccse
plnc
bublebbur
150
805
12:00
1000
(abll antesds
Souaial 1.77
18
Figure 11: Histograms of weight-large

---

Under review as a conference paper at ICLR 2020
20
200
199
300
50
13
100
80-
150
54
775
212
509
6.0
800
1000
Indi gran
22g bords
20K
4000
204
75
Invem))
*oueere dereee
E55
.008
20000
30.00
5.99
1000
300
18000
2048
-004
314
+00
600
3.0
4/4
Straskt
Jock Brathnnttyyyyy cttes
1522
Ycmtal
184
30
3
Isaalil
20
2000
Figure 12: Histograms of weight-small

---

Under review as a conference paper at ICLR 2020
Murkowadd
2-
2.
25
E:4e
Nods
Ang 3ees
2
2+
1.2
3.0
5B2
rdit
Inali rocket
scafed bewk)
2
20
Figure 13: Histograms of confidence

---

Under review as a conference paper at ICLR 2020
Yes 12gg
Yes millxcet-iix
650000
14,000
150000
19:90
180000
0.
24
08
255
20
sralkieod1572
196
200
180
408
kupsiest 476
$800
3000
-006
-4000
1000
26000
46000
10000
090
100000
-2000
1808
Figure 14: X-axis is test (human) and y-axis is prediction by ARD regressors
21


Reviews:
[{"review_id": "SyxG5xWcqB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper attempts to study if learned word embeddings for common objects contain information about \"numerical common sense\". The hypothesis is that certain numerical information may co-occur with the words for certain objects/measurement units within their context windows. To verify this hypotheses, the authors have created a dataset through a crowd-sourcing service which represents \"numerical common sense\". Using this dataset, the authors examine the predict abilities of regressors trained on learned word embeddings and the aforementioned crowd-sourced dataset. The hypothesis is that if the regressors demonstrate good accuracy, then the word embeddings contained information relevant to \"numerical common sense\". To the best of my knowledge, this is the first paper that attempts to analyze learned word embeddings in the context of numerical common sense.\n\nThis paper should be rejected because (1) the NCS datasets are too small to represent \"numerical common sense\" (2) the NCS datasets contain faulty data points and (3) the results from the experiments conducted are not sufficient to accept or refute the hypotheses.\n\nMain argument\n\nThe first question that we must ask is - are the NCS-50x1 and NCS-60x3 datasets reliable for experiments on \"numerical common sense\". No, because of two flaws:\n\n(1) The number of samples in the dataset is too small to represent \"numerical common sense\". Consider the histogram for object \"dog\" in Figure 2. If the largest data point in this plot was absent, the average of the distribution would be smaller by several orders of magnitude. Perhaps there are other objects in the dataset which are missing samples from the tail end of the distribution that could have large effects on the mean of the collected dataset. \n\n(2) Some data points in the dataset don't make sense to me. For example, Fig 2 represents the \"small\" dataset, yet I see samples like 400m long dogs, 40m long cats, 150m long monitors and 20m long mice?\n\nAlso, it is not clear how the confidence scores of the participants were taken into account when training the regressors or if they were used at all.\n\nIf the NCS dataset does not represent \"numerical common sense\", it invalidates all experimental results from the paper.\n\nMy second issue with the paper is that it is not possible to conclude if the experimental results support or refute the hypothesis (ignoring the issue with the dataset):\n\n1. In tables 2 and 3, the correlation coefficients were quite low and and the MAEs were pretty large. In Table 3, rows 1 and 2, even though the correlation is 0.57 and 0.48, the MAE is 100 million yen and 7.5 million yen respectively which is quite large. To me this suggests that just because the correlation is larger we cannot conclude mean that the model is performing well.\n\n2. It is unclear why the correlation coefficient was chosen to decide that ARD is the superior model in experiment 1. The MAE for random forests with concatenated feature vectors was an order of magnitude smaller than that of the ARD model.\n\n3. Why are the correlation coefficients missing for the unit-only experiment in Table 2? The LS model shows very good MAE relative to the other models and perhaps the correlation should have been measured for that as well? In fact, if the correlation coefficients for this case is comparable to the case with concatenated features, it would mean that the word embedding for the object is not helping at all! Moreover, I find it surprising that the LS model with concatenated features performs worse than the unit-only features. We cannot conclude if paper's interpretation about the results is correct unless this missing information is provided.\n\n4. It is hard to judge what a correlation coefficient of 0.57 means. Why didn't you provide a scatter plot of the predictions vs targets as well? It often happens that even noisy plots demonstrate good correlations.\n\n5. The paper should have additional ablation studies - for example, what would happen in the concatenated feature vector experiment if you trained the regressors using randomly initialized word embeddings instead of the trained word embeddings? Do you get the same performance as learned word embeddings?", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "Skl7ei7gcH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper describes the collection and analysis of a numerical common-sense dataset. The paper also states that a novel regression method is presented for quantifying numerical common sense.\n\nStrengths:\n- the numerical common sense task and its related challenges are presented and motivated clearly\n- the data/annotations and their analysis are presented clearly. The experiment can be replicated reasonably (provided that the data is released, as the authors state)\n\nWeaknesses:\n- the dataset is too small (230 pairs of units and values). Two observations follow from this: (1) such small-scale data is of limited use to state of the art (deep learning) data-hungry approaches; (2) the relatively low cost of crowdsourcing typically results in much bigger datasets. \n- some decisions or definitions seem a bit ad hoc and are not convincing. For instance: in Table 1, why is the apple measured in centimetres but the coffee cup measured in meters? Another example from section 1: why are temperature and weight described as nonphysical scales, but money described as a subjective scale? Another example of a definition: \"similar words share similar units and thus concrete words contains numerical information\". Statements like these are problematic because one could easily think of counter examples where this is not the case.\n- the experimental findings are not based on state on the art methods applied in a setup where the robustness and transferability of the methods on other data and domains has been analysed and discussed. \n\nOverall this work is interesting as a starting point. My advice is to consider scaling up both the data and the methods, and to accompany this by a more comprehensive analysis of limitations.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "S1gt14y3YS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I have read the author response.  Thank you for responding to my questions.\n\nThis paper aims to predict typical “common sense” values of quantities using word embeddings.  It includes the construction of a data set and some experiments with regression models.  The general direction of this work is worthy of study, but the paper needs additional justification for its task, better discussion of recent related work, and more development of its regression models.\n\nThe work starts by describing the construction of interesting crowdsourced data sets that include people’s estimates of typical quantities, what they would consider to be low or high values for a given object in given units e.g. the temperature of a hot spring or the height of a giraffe.  Overall, the data sets are interesting but are not especially large (2300 total [low, high] pairs of 230 different quantities).  Further, the particular task formulation here needs more justification.  I think most of us would agree that common sense is a critical AI challenge, and that the question of whether embeddings reflect typical quantities is important.  But, in a paper where the data set is considered a primary contribution, I would expect more justification for exactly how this task is formulated, and which objects and units were selected.  As one example, why ask about “large” and “small” values rather than something with more precise semantics (like the 10th and 90th percentile, for example)?  I also felt that the introduction could be improved to provide more convincing motivation.  E.g., the first paragraph only says that humans apply different adjectives (like “hefty” and “cheap”) to different things depending on their numerical attributes (weight, cost), but does not argue why teaching AI systems to use those adjectives is a priority.\n\nRegarding related work, the paper is missing a discussion of several relevant papers that use embeddings to obtain relative comparisons or estimates of commonsense properties of objects, including:\n\nForbes, Maxwell, and Yejin Choi. \"Verb physics: Relative physical knowledge of actions and objects.\" ACL 2017\n\nYang, Yiben, et al. \"Extracting commonsense properties from embeddings with limited human guidance.\" ACL 2018\n\nElazar, Yanai, et al. \"How Large Are Lions? Inducing Distributions over Quantitative Attributes.\" ACL 2019\n\nThe paper then presents the performance of some regression models.  These models are standard existing techniques, and given the relatively low performance I would have liked more development of the models and more analysis of the performance.  For a conference like ICLR I would expect to see a more thorough exploration and analysis of possible models for the task.  Looking at more powerful neural regressors (perhaps using contextual embeddings rather than just fixed word embeddings) might be one option.  Offering an explanation for why ARD seems to work better than the other approaches would be helpful.\n\nMinor: In Table 3, the way that small and large are interleaved makes it hard to compare systems, I think presenting all the small results together, and large results together may help.  In Figure 2, it would be helpful to see the histogram for size-large within the same plots here, so we could see how far apart they are.\n\n“Because Skip-gram has to handle more words to predict words, we assume Skip-gram will obtain more information about numerical values.”\n-- I didn’t understand what you meant about skip-gram having to “handle more words to predict words.”  Also, I did not understand how this entails that skip-gram would obtain more info about numerical values.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors tackle an interesting and important problem, developing numerical common-sense. They use a crowdsourcing service to collect a dataset and use regression from word embeddings to numerical common sense.

Reviewers were concerned with the size and quality of the dataset, the quality of the prediction methods used, and the analysis of the experimental results.

Given the many concerns, I recommend rejecting the paper, but I encourage the authors to revise the paper to address the concerns and resubmit to another venue.
Decision: Reject