Title: MEASURING NUMERICAL COMMON SENSE: IS A WORD EMBEDDING APPROACH EFFECTIVE?

Abstract: Numerical common sense (e.g., "a person with a height of 2m is very tall") is essential when deploying artificial intelligence (AI) systems in society. To predict ranges of small and large values for a given target noun and unit, previous studies have implemented a rule-based method that processed numeric values appearing in a natural language by using template matching. To obtain numerical knowledge, crawled textual data from web pages are frequently used as the input in the above method. Although this is an important task, few studies have addressed the availability of numerical common sense extracted from corresponding textual information. To this end, we first used a crowdsourcing service to obtain sufficient data for a subjective agreement on numerical common sense. Second, to examine whether common sense is attributed to current word embedding, we examined the performance of a regressor trained on the obtained data. In comparison with humans, the performance of an automatic relevance determination regression model was good, particularly when the unit was yen (a maximum correlation coefficient of 0.57). Although all the regression approach with word embedding does not predict values with high correlation coefficients, this word-embedding method could potentially contribute to construct numerical common sense for AI deployment.

1. INTRODUCTION:

We live in a world where common sense is a basic requirement: if a person's weight is under 50 kg, then they are likely to be very thin; if a person's weight exceeds 80 kg, then they are likely to be hefty. A similar concept can be extended to situations involving money. If a new high-spec laptop is $300, then it is considered to be quite cheap. Therefore, it is essential to account for common sense in future artificial intelligence (AI) systems.

In a society that robots communicate with humans, these robots have to estimate how large the thing is (e.g., If a robot mistakenly estimate the object weight, it may harm humans and/or robots themselves).

There are quantitative units such as physical scales (meter, kilogram, etc.) and money; studies have treated these scales as a unit.

Distributed semantic representations have gained much attention and showed a great performance in varieties of NLP tasks. Mikolov et al. (2013) developed a skip-gram model to obtain semantic representations by using a novel word-embedding approach. Their distributional word embedding exhibited a remarkable performance in analogy tasks (e.g., "King -Man + Woman = Queen").

In more recent studies, Gladkova et al. (2016) prepared varieties of analogy pairs named the Bigger Analogy Test Set (BATS) and compared GloVe (Pennington et al. (2014) ) and count-based (i.e., PPMI + SVD) model (Church & Hanks (1990) ). In our point of view, we assume word themselves have numerical information. That is, similar words share similar units and thus concrete words contains numerical information. The primary motivating assumption is that, similar to analogy, contextual flow in word sequences would have specific repetitive representations, and thus we assume that this specific word representation (e.g., co-occuring unit word such as gram) would include the numerical knowledge for a given target object.

Here, we attempt to analyze the relationship between human-generated and extracted common sense by using word embedding. We assume that common sense is attributed to distributional word embeddings such that a trained regressor can predict numerical values.

The uniqueness in this field lies in treating abstract concepts as common sense. Other than the size of the object, we focus on nonphysical scales, such as temperature and weight, as well as subjective scales, such as money. Moreover, a challenging problem when estimating numerical common sense is that there are few datasets with labeled numerical values. This issue is tackled in our study through a word embedding approach. The contributions we make are two-fold:

• We used a crowdsourcing service to obtain a sufficient number of subjective agreements on numerical common-sense data.

• We examined the predictabilities of regressor trained using word embedding and crowdsourced data. We also examined whether a concatenated semantic vector can effectively predict numerical values.

The remainder of the paper is organized as follows: After introducing related works in Section 2, we describe the collected datasets in Section 3. The results of the experiments conducted using the obtained data are then presented in Section 4. Subsequently, in Section 5, the obtained results are discussed and interpreted. The study is summarized and future scope is described in Section 6.

2. RELATED WORK:

2.1 APPLICATIONS THAT TREAT NUMERICAL VALUES Narisawa et al. (2013) proposed methods that use a large amount of text, related to the abovementioned units, to determine common sense text patterns. Their approach estimates the range of the units co-occurring within a context and examines whether a given value is large, small, or normal based on its distribution. Their approach utilizes textual patterns with which speakers explicitly express their judgment regarding the value of a numerical expression. Similarly, Hayakawa & Hagiwara (2014) used snippets from a web search engine to extract more scalable but a lesser amount of text data, related to the units, through the design of heuristic functions that enable the system to handle input frequencies, which are often very noisy. These two approaches require careful intuitions when determining the thresholds of a proposed method or building an effective function for each numerical value. Qin et al. (2018) further developed a numerical variable extraction mechanism. They modeled the joint probability of data fields, texts, phrasal spans, and latent annotations with an adapted semi-hidden Markov model and imposed a soft statistical constraint to further improve the performance of the model.

Another approach focuses on using different modalities. A 165 cm human cannot be taller than a 1 km long bridge. This concept has a wider range of applications. Bagherinezhad et al. (2016) considered object sizes using a computer vision technique.

In the context of only text, particularly in relation to numerical common-sense extraction, most methods rely on large amount of data with carefully designed heuristics. As such, it would be beneficial if it was possible to extract common sense only using certain human evaluations.

Forbes et al. Forbes & Choi (2017) present an approach to infer relative physical knowledge of actions and objects along five dimensions (e.g., size, weight, and strength) from unstructured natural language text. They reported that it is possible to extract knowledge of actions and objects from language and that joint inference over different types of knowledge improves performance. Yan et al. Yang et al. (2018) proposed and assessed methods for extracting one type of commonsense knowledge, object-property comparisons, from pre-trained embeddings. They also showed that an active learning approach that synthesizes common-sense queries can boost accuracy. 

2.3. THEORETICAL BACKGROUND ON NUMERICAL WORD EMBEDDING:

We describe why numerical values are related to word embedding. In the context of distributed word embeddings, "count-base" and "prediction-based" method has something in common. As a traditional count-based method, before conducting SVD, matrices of Positive Pointwise Mutual Information (PPMI) is used to obtain word embeddings (Church & Hanks (1990) ). More concretely, PPMI = max(0, PMI(x, y)) = max 0, log 2 P (x, y) P (x), P (y)

(1)

where P (x) and P (y) are frequencies of each word and P (x, y) indicates co-occurrence of the two words. As such, a word that co-occur with words in a group and another word that co-occur with words in the smilar group will have similar embedding. Numerical representation will on this point will be obtained in this case.

Similarly, for current prediction-based word embeddings, a loss fucntion of continuous bag-of-words (CBOW) is represented as:

EQUATION

where w t-1 , w t+1 are to predict w t , resulting in surrounding words will have similar embedding. In addition, loss function of Skip-gram is calculated as:

EQUATION

where w t is to predict w t-1 , w t+1 . Because Skip-gram has to handle one word to predict multiple words, we assume Skip-gram will obtain more information about numerical values.

3. DATA COLLECTION:

We used the Japanese crowdsourcing service Lancers to collect data on numerical common sense1 . Table 1 lists examples of objects and units for each dataset.

We asked participants to annotate these objects and units to indicate "from which value you think it's large" and "from which value you think it's small." Note that for temperature, this "large quantity" will be "high temperature." For Numerical Common Sense including 50 items with one pair (NCS-50x1), we asked 10 participants to create 50 pairs of samples with different units. For Numerical Common Sense including 60 items with three pairs (NCS-60x3), we asked 10 different participants to construct 60 pairs of samples with three different units. In a total, we collected 50 + 60 x 3 = 230 pairs of units and values. We also collected participants' confidence (5-rated evaluation: 1 is least confident and 5 is most confident) for their answers.

3.1. CONSTRUCTION OF NCS-50X1:

In NCS-50x1, we selected terms that represent an "object" and "unit" based on the results obtained by Hayakawa & Hagiwara (2014) and included all their 26 examples. Moreover, we added other 24 object and unit pairs. This dataset was collected on February 28, 2019.

Figure 1 illustrates some examples that were collected. The ranges of "small" and "large" differ depending on the term. For example, large in train transition is around 10 times more (with a large variance) than small, while both large and small in body temperature have smaller variances. Moreover, there are terms whose confidence from participants is more than others. Compared with guessing large/small of temperatures in hot spring, participants are confident when predicting large/small of sleeping hours.

3.2. CONSTRUCTION OF NCS-60X3:

In NCS-60x3, for a more exhaustive study on the variation concept, we limit objects and units using WordNet Miller et al. (1990) and brain-based knowledge representation Naselaris et al. (2009) as a reference. In the supplementary material provided by Naselaris et al. (2009) , a handful of word concepts to check the brain decode and encode predictabilities were shown; we assumed these concepts could be applied to crowdsourcing on a reasonable scale. We use 60 objects considering the above concepts and the obtained crowdsourced data. Particularly, animate, artificial, and food objects were taken into account. This dataset was collected on March 3, 2019.

We depict the histogram of 18 example items in Figure 2 and 3 .

As the figure shown, the results from participants are very diverged. To make a fair comparison, we use the same unit through size, and weight (i.e., m is converted to cm by computing "x 100"). We analyze regressors trained on the word embeddings along with human evaluated data to examine whether common sense is attributed to current word embedding. Figure 4 shows a diagram of this experiment. After checking performance on various units with/without word embedding concatenation on different regressor in Experiment 1, we compare the regression performance on each word embedding type trained on different corpus. Then we also compare that performance with humans in Experiment 2. We intend to 1. investigate overall tendencies using NCS-50x1 in the first experiment because we include terms (objects and units) from one of the pioneer works of numerical common sense (Hayakawa & Hagiwara (2014) ), and 2. obtain more detailed points including results on selected units and types of word embeddings comparing with human participants in the second experiment.

Section 3.1 provides more details on the experimental settings and Section 3.2 describes experiments 1 and 2, where we use NCS-50x1 and NCS-60x3, respectively.

4.1. EXPERIMENTAL SETTINGS:

We describe the details about word embeddings and machine learning algorithms that we use. We followed the procedure described below to prepare for the experiment.

• We use word embeddings trained on Japanese Wikipedia, which was downloaded in Feb 2015. To simplify the problem, when there were multiple words in an object, feature vectors were averaged. Note that when tokenizing, we used MeCab (Kudo et al. (2004) ). We then obtained word embeddings based on a Continuous Bag-of-Words (CBOW) model with 200 dimensions. Henceforth, we refer to this as "w2v-wiki." • We use fastText word embeddings Bojanowski et al. (2017) trained on Japanese Wikipedia that are available at Grave et al. (2018) . The model was trained using CBOW with positionweights, in dimension 300, and character n-grams of length 5, a window of size 5 and 10 negatives. Henceforth, we refer to this setting as "fT-wiki." • We use word embeddings trained on crawled Japanese web texts (Yata) , Nihongo Web Corpus (NWC). The texts are composed of approximately 100 million html documents (3.25TB in total), which resulted in 396 GB of plain texts. Note that when tokenizing, we used MeCab with the NEologd dictionary. The model was trained using CBOW, in dimension 500, with a window of size 15. We refer to this setting as "w2v-nwc." • For experiment 1, we prepared different types of word vectors; object only, unit only, and concatenated feature, respectively. For object only (e.g. "suimin": sleeping) and unit only (e.g., "jikan": hours), word embeddings are directly extracted from "w2v-wiki" and regressors are trained and tested. For concatenated features, those two word vectors (namely object and unit) were concatenated, which resulted in 400 dimensions for the concatenated features. For Experiment 2 where we use NCS-60x3 dataset, we also use "fT-wiki" whose concatenated embedding dimension is 600, and "w2v-nwc" whose one is 1000. • We employed a linear regressor with the least square (LS) error penalty and a linear regressor with an automatic relevance determination (ARD) prior. We also use random forest (RF) regressor for comparison. In the RF, the number of decision trees was set to 10; in ARD, the parameters were set to alpha1=1e-06, alpha2=1e-06, lambda1=1e-06, and lambda2=1e-06. The LR input will be normalized before regression by subtracting the mean and dividing by the l2-norm. After average numerical values were obtained from 10 subjects, we conducted a 10-fold cross validation. To examine performances of those regressors, we use the mean absolute error (MAE) and the Pearson's correlation coefficient. • To compare the predictability between CBOW and Skip-gram, and increament of word embedding dimensions, we use Japanese Wikipedia and change the dimensions from 100 to 500 in experiment2. • We used gensim Rehurek & Sojka (2010) for training word embeddings and feature extraction, and scikit-learn Pedregosa et al. (2011) for executing machine learning algorithms. We use w2v-wiki for Experiment 1 and all three word embeddings for Experiment 2.

4.2. EXPERIMENTAL RESULTS:



4.2.1. EXPERIMENT 1 ON NCS-50X1:

The regressor performance results are shown in Table 2 . Essentially, MAE showed high prediction errors and the Pearson's correlation coefficient was quite low. The Concat feat. in ARD appeared to have a higher correlation than other non-human methods. Therefore, in the next experiment we employ word embedding concatenation.

4.2.2. EXPERIMENT 2 ON NCS-60X3:

Table 3 list the results for 60 objects with three different fixed units obtained from linear regressors.

Here, we fixed a unit and examined the predictabilities of word vectors trained on human evaluation. In Experiment 2, we also compared the results of the trained regressor with those of human evaluation.

Humans scores and predictors are calculated in the same manner. Concretely, in a 10-fold cross validation manner, correlation between evaluation scores from each subject and averaged other 9 subjects were calculated. Humans performed better than both of regressors, yet ARD had a correlation coefficient of 0.57, which appeared to be comparable to that of humans when the unit is Yen and gram.

In most cases, the regressor had better prediction in MAE when the unit was centimeter. This may be because there is less variation in values. In contrast, the correlation coefficient of both regressors showed better performance for the unit of gram.

In addition, we examined CBOW and Skip-gram with an increase in dimensions. When we tested the data on Japanese Wikipedia, we observed that the accuracy changed with an increase in the dimensions For ADR, the average correlation coefficient across units [Yen, g, cm] was 0. 

5. DISCUSSION:

We computed the predictabilities of numerical values solely from word embeddings to examine whether they have such information. This idea is based on whether corpus has some contextual information of numerical values. Because word embedding models treat context windows, when predictabilities are obtained, some kind of numerical knowledge is embedded as co-occurrence. Word embedding is treating co-occurrence and numerical values may be included. In context, objects that are occur is similar and then, text embedding will be possible because the numerical tendencies will be similar. One can imagine that w2v-nwc embedding outperformed in correlation coefficient because of not only its size and diversity, but also contextual window size.

Overall, predicting large was quite diverged on both on regressors and humans.

As indicated by Figure 1 , numerical values from annotators vary for specific words. As depicted in Figure 2 , some metrics especially size had a large variance. For example, the size of some items (e.g., "plastic model" and "audio system") differed more than 1000 times depending on persons when treating large. Basically, it was also difficult to predict money when it comes to MAE. A few unknown but expensive objects such as "private jets" considerably influenced on the result. While our method on MAE did not work well, correlations between predicted numerical values and human evaluations were observed. Prediction by regressor is quite comparable to human in terms of correlation coefficient. As we anticipated, Skip-gram showed a better performance.

6. CONCLUSION:

We are living in a world where we need common sense; thus quantifying numerical common sense is essential to deploy AI systems in society. We collected and analyzed numerical common-sense values through a crowdsourcing web service. Our contributions are two-fold.

First, we collected data on a subjective yet not fully explored area. We then compared current approaches from a quantitative perspective. To this end, we first used a crowdsourcing service to obtain sufficient data for a subjective agreement on numerical common sense. We constructed two datasets, namely, NCS-50x1 and NCS-60x3.

Second, we compared the averaged numerical values from participants on the crowdsourcing platform with that of the regressor using word vectors. To the best of our knowledge, this is the first approach that tackles this point. In comparison with humans, the performance of an automatic relevance determination regression model was good, particularly when the unit was yen (up to a correlation coefficient of 0.57). In addition, predictability increased when we employed the Skip-gram.

Even though we confirmed positive correlation coefficients, the model was not comparable with the data collected using the crowdsourcing service when it was trained on different units (e.g., when trained on yen and tested on gram).

Although our novel regression approach with word embedding does not always predict values with high correlation coefficients, it exhibited potential to quantify numerical common sense.

In addition, obtaining real numerical distributions would be very helpful in advancing this area.

We have a number of large copora with numerical numbers. For example, Wikidata, which was converted from Wikipedia to be precessed by computers can be useful as an extention. We believe those datasets can be utilized to expand the size of the numerical common sense dataset. Moreover, it would be more effective to employ more powerful newural regressors.

Simultaneously, we plan to pursue the sophistication of semantic representations of the model. To check the agreement on each participant, we conducted a correlation analysis as shown in Figure 5 .

To calculate the inter-rater agreements, we aggregated the scores from participants, and we computed correlation between groups of subjects. In the figure, the X-axis indicates the number of subjects and the Y-axis is the calculated correlation. The bars indicate the standard deviation of the correlation. As the number of the participants increases, reasonably high correlation is observed.

Compared with large and confidence, small had higher correlation.

A.1.2 NCS-60X3:

The results of a correlation analysis are shown in Figure 6 when the number of participants increases.

In the figure, the X-axis indicates the number of subjects and the Y-axis is the calculated correlation.

The bars indicate standard deviation of the correlation. Except for size large, as the number of participants increases, higher correlations were obtained. Because a few items (e.g., "plastic model" and "audio system") that are not common for participants were included, it seems that estimating sizes, especially large was quite diverged. Compared with other modalities, prices shared similar tendencies, which resulted in a higher correlation.

Herein, we list results of histograms. Note that these represent values in NCS-60x3 dataset and showing "price-large", "price-small", "size-large", "size-small", "weight-large", "weight-small." and confidence with 5-point scales. Units (which corresponds to the X-axis) are "Yen", "Yen", "cm", "cm", "g", "g", and "1-5" respectively for each page in all figures. Note that the number of the participants is 10 so that the maximum number of the Y-axis will be 10 at the maximum. Figure 7 , 8, 9 10, 10, 11, 12, and 13 show histograms of price-large, price-small, size-large, size-small, weight-large, weight-small, and confidence, respectively. We show a scatter plot of test human and prediction in Figure14. Note that we conducted the 10 fold-cross validation for prediction. 

:

We plan to make the dataset available to the public after publication

Figures and Tables:

Figure 1: Some examples from NCS-50x1. The histogram indicates the items' averages and the bars represent the standard deviations.

Figure 2: Histogram of money-small on 18 out of 60 items. The X-axis is Yen and the Y-axis represents number of participants.

Figure 3: Histogram of size-small on 18 out of 60 items. The X-axis is centimeter and the Y-axis represents number of participants.

Figure 4: Diagram of experimental procedure

Figure 5: Inter-rater agreements of NCS-50x1

Figure 6: Inter-rater agreements of NCS-60x3

Figure 13: Histograms of confidence 20

web data, and use it to create a new, large resource consisting of distributions over physical quantities associated with objects, adjectives, and verbs.

Table content: <table><tr><td>NCS-50x1</td><td>object</td><td>unit</td><td>NCS-60x3</td><td>object</td><td>unit</td></tr><tr><td colspan="2">family member</td><td>number</td><td colspan="2">laptop</td><td>yen</td></tr><tr><td>milk</td><td/><td>ml</td><td colspan="2">apple</td><td>centimeter</td></tr><tr><td colspan="2">winter climate</td><td>degree Celsius</td><td colspan="2">coffee cup</td><td>meter</td></tr><tr><td colspan="3">university entrance rate percentage</td><td colspan="2">piano</td><td>kilogram</td></tr><tr><td colspan="2">annual income</td><td>yen</td><td colspan="3">private jet plane gram</td></tr><tr><td/><td/><td/><td>wine</td><td/><td/></tr><tr><td colspan="6">Table 1: Examples of objects and units for each dataset. Note that the object and unit need not</td></tr><tr><td>correspond.</td><td/><td/><td/><td/><td/></tr><tr><td colspan="4">2.2 STUDIES ON DISTRIBUTED SEMANTIC REPRESENTATION</td><td/><td/></tr><tr><td colspan="6">The approach proposed herein is closely related to existing semantic vector models (i.e., word em-</td></tr><tr><td colspan="6">bedding). More recent studies used global contexts Pennington et al. (2014) or incorporated other</td></tr><tr><td colspan="6">external data, such as brain activity from fMRI data Fyshe et al. (2014), to improve performance.</td></tr><tr><td colspan="6">fastText Bojanowski et al. (2017) improves performance by considering subwords in the model.</td></tr><tr><td colspan="6">Jeawak et al. (2019) imcoporated geographical information into semantic representations to deal</td></tr><tr><td colspan="2">with locations for flicker images.</td><td/><td/><td/><td/></tr></table>

Results obtained from linear and random forest regressors. Note that Corr. Coef. indicates Pearson's correlation coefficient and MAE is mean absolute error

Table content: <table><tr><td>object only</td><td colspan="2">Mean Abs. Err. Corr. Coef.</td></tr><tr><td>ARD</td><td>1.20 x 10 7</td><td>0.080</td></tr><tr><td>LS</td><td>1.15 x 10 7</td><td>0.011</td></tr><tr><td>RF</td><td>1.04 x 10 7</td><td>0.002</td></tr><tr><td>unit only</td><td/><td/></tr><tr><td>ARD</td><td>1.20 x 10 7</td><td>N/A</td></tr><tr><td>LS</td><td>1.10 x 10 1</td><td>N/A</td></tr><tr><td>RF</td><td>9.91 x 10 6</td><td>N/A</td></tr><tr><td>Concat. feat.</td><td/><td/></tr><tr><td>ARD</td><td>1.20 x 10 7</td><td>0.254</td></tr><tr><td>LS</td><td>1.30 x 10 7</td><td>-0.035</td></tr><tr><td>RF</td><td>9.08 x 10 5</td><td>0.181</td></tr></table>

Mean absolute error results for 60 objects with 3 different fixed units obtained from linear and random forest regressors with unit on the left (Yen, cm, and g). Pearson's correlation coefficient results on the right. Bold digits in each row indicate higher performance. when using Skip-gram for ARD regression. Average correlation coefficient across units was 0.343 with200-d, 0.378 with 300-d, 0.325 with 400-d, and 0.372 with 500-d.

Table content: <table><tr><td/><td/><td>MAE</td><td/><td/><td>Corr. Coef.</td><td/></tr><tr><td>unit: YEN</td><td>w2v-wiki</td><td>fT-wiki</td><td>w2v-nwc</td><td colspan="3">w2v-wiki fT-wiki w2v-nwc</td></tr><tr><td>large on ARD</td><td colspan="4">9.07 x 10 7 9.07 x 10 7 9.07 x 10 7 0.360</td><td>-0.237</td><td>0.571</td></tr><tr><td>small on ARD</td><td colspan="4">7.56 x 10 6 7.56 x 10 6 7.56 x 10 6 0.403</td><td>0.013</td><td>0.418</td></tr><tr><td>large on LS</td><td>2.89 x 10 8</td><td>3.80 x 10 8</td><td>1.85 x 10 8</td><td>0.216</td><td>-0.033</td><td>0.444</td></tr><tr><td>small on LS</td><td>2.45 x 10 7</td><td>3.39 x 10 7</td><td>2.45 x 10 7</td><td>0.212</td><td>-0.118</td><td>0.268</td></tr><tr><td>large on RF</td><td>1.38 x 10 8</td><td>7.56 x 10 7</td><td>1.49 x 10 8</td><td>0.127</td><td>0.217</td><td>0.500</td></tr><tr><td>small on RF</td><td>1.14 x 10 7</td><td>4.40 x 10 6</td><td>1.29 x 10 7</td><td>0.119</td><td>0.262</td><td>-0.012</td></tr><tr><td>large on human</td><td/><td>6.08 x 10 7</td><td/><td/><td>0.753</td><td/></tr><tr><td>small on human</td><td/><td>4.73 x 10 6</td><td/><td/><td>0.894</td><td/></tr><tr><td>unit: cm</td><td/><td/><td/><td/><td/><td/></tr><tr><td>large on ARD</td><td>712</td><td>911</td><td>695</td><td>0.120</td><td>0.0070</td><td>0.310</td></tr><tr><td>small on ARD</td><td>51.4</td><td>189</td><td>142</td><td>0.205</td><td>-0.018</td><td>0.487</td></tr><tr><td>large on LS</td><td>383</td><td>793</td><td>479</td><td>0.502</td><td>0.111</td><td>0.249</td></tr><tr><td>small on LS</td><td>31.9</td><td>284</td><td>142</td><td>0.270</td><td>-0.014</td><td>0.460</td></tr><tr><td>large on RF</td><td>339</td><td>189</td><td>117</td><td>0.236</td><td>0.259</td><td>0.067</td></tr><tr><td>small on RF</td><td>25.1</td><td>105</td><td>102</td><td>0.148</td><td>0.162</td><td>0.234</td></tr><tr><td>large on human</td><td/><td>312</td><td/><td/><td>0.828</td><td/></tr><tr><td>small on human</td><td/><td>51.9</td><td/><td/><td>0.905</td><td/></tr><tr><td>unit: g</td><td/><td/><td/><td/><td/><td/></tr><tr><td>large on ARD</td><td colspan="2">3.99 x 10 4 3.98 x 10 7</td><td>3.98 x 10 7</td><td>0.508</td><td>-0.371</td><td>0.359</td></tr><tr><td>small on ARD</td><td colspan="2">9.21 x 10 3 9.25 x 10 6</td><td colspan="2">9.25 x 10 6 0.339</td><td>-0.265</td><td>0.358</td></tr><tr><td>large on LS</td><td>1.27 x 10 5</td><td>2.00 x 10 8</td><td>8.33 x 10 7</td><td>0.180</td><td>0.182</td><td>0.118</td></tr><tr><td>small on LS</td><td>2.08 x 10 4</td><td>2.56 x 10 7</td><td>1.47 x 10 7</td><td>-0.075</td><td>-0.266</td><td>0.060</td></tr><tr><td>large on RF</td><td>5.52 x 10 5</td><td colspan="3">2.60 x 10 7 1.46 x 10 6 -0.047</td><td>-0.041</td><td>0.041</td></tr><tr><td>small on RF</td><td>1.20 x 10 4</td><td colspan="2">8.61 x 10 6 1.29 x 10 7</td><td>0.292</td><td>0.092</td><td>0.288</td></tr><tr><td>large on human</td><td/><td>3.84 x 10 7</td><td/><td/><td>0.746</td><td/></tr><tr><td>small on human</td><td/><td>8.41 x 10 6</td><td/><td/><td>0.774</td><td/></tr></table>

Peer review (iclr2020):
This paper attempts to study if learned word embeddings for common objects contain information about *numerical common sense*. The hypothesis is that certain numerical information may co-occur with the words for certain objects/measurement units within their context windows. To verify this hypotheses, the authors have created a dataset through a crowd-sourcing service which represents *numerical common sense*. Using this dataset, the authors examine the predict abilities of regressors trained on learned word embeddings and the aforementioned crowd-sourced dataset. The hypothesis is that if the regressors demonstrate good accuracy, then the word embeddings contained information relevant to *numerical common sense*. To the best of my knowledge, this is the first paper that attempts to analyze learned word embeddings in the context of numerical common sense. This paper should be rejected because (1) the NCS datasets are too small to represent *numerical common sense* (2) the NCS datasets contain faulty data points and (3) the results from the experiments conducted are not sufficient to accept or refute the hypotheses. Main argument The first question that we must ask is - are the NCS-50x1 and NCS-60x3 datasets reliable for experiments on *numerical common sense*. No, because of two flaws: (1) The number of samples in the dataset is too small to represent *numerical common sense*. Consider the histogram for object *dog* in Figure 2. If the largest data point in this plot was absent, the average of the distribution would be smaller by several orders of magnitude. Perhaps there are other objects in the dataset which are missing samples from the tail end of the distribution that could have large effects on the mean of the collected dataset. (2) Some data points in the dataset don*t make sense to me. For example, Fig 2 represents the *small* dataset, yet I see samples like 400m long dogs, 40m long cats, 150m long monitors and 20m long mice? Also, it is not clear how the confidence scores of the participants were taken into account when training the regressors or if they were used at all. If the NCS dataset does not represent *numerical common sense*, it invalidates all experimental results from the paper. My second issue with the paper is that it is not possible to conclude if the experimental results support or refute the hypothesis (ignoring the issue with the dataset): 1. In tables 2 and 3, the correlation coefficients were quite low and and the MAEs were pretty large. In Table 3, rows 1 and 2, even though the correlation is 0.57 and 0.48, the MAE is 100 million yen and 7.5 million yen respectively which is quite large. To me this suggests that just because the correlation is larger we cannot conclude mean that the model is performing well. 2. It is unclear why the correlation coefficient was chosen to decide that ARD is the superior model in experiment 1. The MAE for random forests with concatenated feature vectors was an order of magnitude smaller than that of the ARD model. 3. Why are the correlation coefficients missing for the unit-only experiment in Table 2? The LS model shows very good MAE relative to the other models and perhaps the correlation should have been measured for that as well? In fact, if the correlation coefficients for this case is comparable to the case with concatenated features, it would mean that the word embedding for the object is not helping at all! Moreover, I find it surprising that the LS model with concatenated features performs worse than the unit-only features. We cannot conclude if paper*s interpretation about the results is correct unless this missing information is provided. 4. It is hard to judge what a correlation coefficient of 0.57 means. Why didn*t you provide a scatter plot of the predictions vs targets as well? It often happens that even noisy plots demonstrate good correlations. 5. The paper should have additional ablation studies - for example, what would happen in the concatenated feature vector experiment if you trained the regressors using randomly initialized word embeddings instead of the trained word embeddings? Do you get the same performance as learned word embeddings?