Learning Numeral Embedding
Authors: Chengyue Jiang
Zhonglin Nian
Kaihao Guo
Shanbo Chu
Yinggong Zhao
Libin Shen
Kewei Tu
Venue:  (2020)
Keywords: Natural Language Processing
Numeral Embedding
Word Embedding
Out-of-vocabulary Problem

Abstract: Word embedding is an essential building block for deep learning methods for natural language processing. Although word embedding has been extensively studied over the years, the problem of how to effectively embed numerals, a special subset of words, is still underexplored. Existing word embedding methods do not learn numeral embeddings well because there are an infinite number of numerals and their individual appearances in training corpora are highly scarce.
In this paper, we propose two novel numeral embedding methods that can handle the out-of-vocabulary (OOV) problem for numerals. We first induce a finite set of prototype numerals using either a self-organizing map or a Gaussian mixture model. We then represent the embedding of a numeral as a weighted average of the prototype number embeddings. Numeral embeddings represented in this manner can be plugged into existing word embedding learning approaches such as skip-gram for training.
We evaluated our methods and showed its effectiveness on four intrinsic and extrinsic tasks: word similarity, embedding numeracy, numeral prediction, and sequence labeling. 

Under review as a conference paper at ICLR 2020
LEARNING NUMERAL EMBEDDING
Anonymous authors Paper under double-blind review
ABSTRACT
Word embedding is an essential building block for deep learning methods for nat- ural language processing. Although word embedding has been extensively studied over the years, the problem of how to effectively embed numerals, a special subset of words, is still underexplored. Existing word embedding methods do not learn numeral embeddings well because there are an infinite number of numerals and their individual appearances in training corpora are highly scarce. In this paper, we propose two novel numeral embedding methods that can handle the out-of- vocabulary (OOV) problem for numerals. We first induce finite set of prototype numerals using either a orgf-organi map or a Gaussian mixture model. We then represent the embedding of numeral weighted average of the proto- type number embeddings. Numeral embeddings represented in this manner can be plugged into existing word embedding learning approaches such skip-gram for training. We evaluated our methods and showed its effectiveness on four intrin- sic and extrinsic tasks: word similarity, embedding numeracy, numeral prediction, and sequence labeling.
INTRODUCTION
Word embeddings, the distributed vector representations of words, have become the essential build- ing block for deep learning approaches to natural language processing (NLP). The quality of pre- trained word embeddings has been shown to significantly impact the performance of neural ap- proaches to a variety of NLP tasks. Over the past two decades, significant progress has been made in the development of word embedding techniques (Lund & Burgess] 1996; Bengio et al., 2003 Bullinaria Levy 2007 Mikolov et al., 20136] Pennington et al.] 2014). However, existing word embedding methods do not handle numerals adequately and cannot directly encode the numeracy and magnitude of a numeral (Naik et al. 2019). Most methods have a limited vocabulary size and therefore can only represent small subset of the infinite number of numerals. Furthermore, most numerals have very scarce appearances training corpora and therefore are more likely to be out- of of-vocabulary (OOV) compared non nom-numerical words. For example, numerals account for 6. 6.1% of all unique tokens in English Wikipedia, but in Glo Penniigton et al. (2014) which is partially trained on Wikipedia, only 3.79% of its vocabulary is numerals. Previous work (Spithourakis al.) 2016) also shows that the numeral OOV problem is even more severe when learning word embed- dings from corpora with abundant numerals such clinical reports. Even if a numeral is included in the vocabulary, its scarcity in the training corpus would negatively impact the learning accuracy of its embedding.
The inadequate handling of numerals in existing word embedding methods can be problematic in scenarios where numerals convey critical information. Take the following sentences for example,
"Jeff is 190, so he should wear size XXL" (190 is reasonable height for size XXL)
"Jeff is 160, so he should wear size XXL" (160 is an unreasonable height for size XXL.
"Jeff is 10, so he should wear size XS" (10 is an age instead of a height)
If the numerals in the example OOV their embeddings not accurately learned, then it becomes impossible to judge the categories of the numerals or the reasonableness of the sentences.
In this paper, propose two novel methods that can produce reasonable embeddings for any nu- merals. The key idea is to represent the embedding of a numeral as a weighted average of a small set of prototype number embeddings. The prototype numerals are induced from the training corpus

---

Under review as a conference paper at ICLR 2020
using either self-organizing map (Kohonen 1990) computed based on the differences between the target numeral and the prototype numerals, reflecting the inductive bias that numerals with similar quamtities are likely to convey similar semantic infor- mation and thus should have similar embeddings. Numeral embeddings represented in this manner can then be plugged into traditional word embedding method for training. We empirically evaluate our methods on four tasks: word similarity, embedding numeracy, numeral prediction, and sequence labeling. The results show that our methods can produce high-quality embeddings for both numerals and non- numerical words and improve the performance of downstream tasks.
Gaussian mixture model. The weights
RELATED WORK
Word Embedding Word embeddings are vector representations of words which carry semantic meanings implicitly and are trained without supervision. Most existing word embedding train- ing methods can be divided into two classes. The first class of methods (Lund & Burgess 1996]
Rohde et al., 2006 Bullinaria & Levy 2007 Lebret & Lebret, 2013) extract word co-occurrence statistics from the training corpus, compute word-word matrix based on measures such as PPMI, and then apply dimension reduction techniques such as principle component analysis to produce a low-dimensional vector representation for each word. The second class of methods (Bengio et al.) 2003 Collobert & Weston 2008; Mikolov et al. 2013a b) use a simple neural network to model the relation between word and its context within sliding window in the training corpus. Glo (Pen-) nington et al.] 2014) has been proposed as a method that combines the advantages of both classes. All the above methods have finite vocabulary size and use 'UNK' symbol to represent OOV words. Recent work (Naik et al. (219)) shows that these popular methods do not handle numerals adequately. WWllace al. (2019) shows that existing word embedding methods can encode numer- acy implicitly for high-frequency numerals, but the embedding's 's numeracy for OOV numerals is not investigated. Our goal is to design better numeral embedding methods that can be integrated into traditional word embedding methods and handle the OOV problem for numerals.
Numeracy in natural language Numeral understanding has been found important in textual en- De Marneffe et al., 2008: Roy et al.) 2015) and information extraction Madaan et al., 2016),, but existing systems often use manually defined task-specific features and logic rules to identify numerals, which is hard to generalize to other tasks. A lot of research has been done trying to solve math problems. using either manually designed fea- tures and rules (Roy et al. 2015) Mitra & Baral 2016: Roy & Roth 2016) Upadhyay et al.] (016) or grquence-to-equenee
tailment (Lev et al., 2004
(Inxxaurrondo al.
2015
neural networks Wang et al. (2017), but the quantity numerals is not important in this task and hence existing methods often replace numerals by dummy symbols such as n and 722. SSpithourakis & Riedel (2018) studied different strategies to better model numerals in language models. Chen et al. (2019) created Numeracy-600K dataset and studied the ability of neural network models to learn numeracy. Our work differs from previous work in that we aim to produce general purppose numeral embeddings that can be employed in any neural NLP approach.
3 METHODS
Given a training corpus C, we first extract all the numerals using regular expressions and form dataset X containing all the numbers represented by these numerals. A number (e.g.. 2000) may appear for multiple times in X if its corresponding numerals (e.g., '2000', '2,000', etc.) appear for multiple times in C. We then induce a finite set IP of typical numerals (i.e., prototypes) from X using self-organizing map (Kohonen 1990) a Gaussian mixture model. We also define function sim(ni, n22) outputting the similarity between two arbitrary numbers n, and n2. Now we represent the embedding of any target numeral n weighted average of the prototype number embeddings with the weights computed by the similarity function:
e(n)
Σ a (n.( e(p), PEP
Σ a Re-sim((.)) p) =
(1)
PEP
We use e(.) to denote the embedding of number a is the normalization factor. This formulation satisfies the intuition that numerals with similar quantities are likely to convey similar semantic information and thus should have similar embeddings.
2

---

Under review as a conference paper at ICLR 2020
Our numeral embeddings can be integrated into traditional word embedding methods such skip- gram for training. During training, we back-propagat the error gradient to update the prototype number embeddings. In this way, the prototype number embeddings (and hence all the numeral emheddings) are learned jointly with non-numerical word embeddings.
3.1 SQUASHING NUMBERS TO LOG-SPACE
Inspired by psychological evidence that our brain compresses large quantities nonlinearly using a logarithmic scale on the mental number line (Nieder & Miller] 2003) [ we design the following squashing function to transform all the numbers X into the log-space before prototype induction. Alternatively, we can apply the function only in the similarity function. Besides the psychological motivation, squashing also necessary for our methods to avoid overflow during training when there are very large numbers such 10 in the training corpus.
log(z) + 1,
if x > 1 if x E [-1,1] log(-x)
f(x)
(2)
x <-1
3.2 PROTOTYPE INDUCTION
We develop two methods for inducing small prototype set P from the number dataset X. Denote the number of prototypes by m.
Self-Organd Seff-Organizing Map A self-organizing map (SOM) (Kohonen 1990) is artificial neural net- work that can be viewed as a clustering method. After training a SOM on the dataset X, we regard each cluster centroid as a prototype. One advantage of using a SOM in comparison with traditional clustering methods is that it distributes prototypes more evenly on the number line and may as- sign prototypes number ranges with few training samples, which we expect would lead to better generalizability.
Gaussian Mixture Model Inspired by psychological study of the mental number line (Dehaene eet al., (003) and previous work on language modeling (Spithourakis & Riedel (018),, we train Gaussian mixture model (GMM) to induce number prototypes. GMM defined as follows.
= = = Σ P(z= n| = =
Σ (n;
(3)
1=1
where is latent variable representing the mixture component for random variable U, and A is the probability density function of a normal distribution, and Th, M4g a E R represent the mixing coefficient, mean and standard deviation of the k-th Gaussian component. We train a GMM on the number dataset X using the expectation maaimiizaiiiiiinnnnmmm (EM) or Hard-EM algorithm and regard the means of the learned Gaussian components as our prototypes GMM initialization methods described in Appendix A
{ur,-.. .pm). We use three
3.3 SIMILARITY FUNCTION
For SOM-induced prototypes, we define the following similarity function:
sim(p, n) lg(p) g(m)
B > P
(4)
where function is equal to the squashing function f defined in Eq if we do not apply log transfor- mation before prototype induction and the identity function otherwise. / is hyper-parameter set to 1.0 by default.
For GMM-induced prototypes, we can naturally use the posterior probability of the component assignment to define the similarity function.
T (n:
sim(pk,n) x P(Z = k(U = n)
Pk E
(5)
TWN (1
3

---

Under review as a conference paper at ICLR 2020
weighted average
6000
10000 7-1
00000 of
sim(190, p 1000 P P
E
E,
M
one-hot
1000000
1000
he
i8
190 P (pI - Pm
X-1
X
x1+1
Figure 1: The computational graph when the center word is 'is' and the context words are 'he' and the numeral '190'. We look up the embedding vectors of non-numerical words directly from the embedding matrices and use the weighted average of prototype embeddings as the numeral embedding. Negative sampling is not shown in the figure.
3.4 EMBEDDING TRAINING
We now describe how to integrate our numeral embeddings into traditional word embedding meth- ods for training. We choose skip-gram with negative sampling (Mikolov et al. [2013a b) as the word embedding method here, but many other word embedding methods such as CBOW (Mikolov et al., 2013a), HAL (Lund & Burgess 1996) and GloVe (Pennington et al. 2014) can be used as well.
Skip-gram is a word embedding method based on the idea of context word prediction. The training corpus C' is regarded as sequence of words (I1,.... ZT). ). For token Iı. define the preceding and following e tokens as the context of it- Skip-gram aims to maximize p(*e+ glat) (-c S & c). the probability of context word given the center word 24z To formulate ((2++)124)) skip-gram associates each word with vector representations: the input embedding v², for being center word and the output embedding 2 for being a context word. The input and output embeddings of all the words in the vocabulary V constitute matrices E₁ DD and Fo E Dx|VI respec- tively, where D is the dimension of word embeddings. The conditional probability p( is then defined to based the dot product sz))+)lz;) ed "r,,y" Nagative sampling is used to approximate the normalization factor for the conditional probability.
log p( xx) xe log of
T
Ux
E (a(ar
log (-oo
T T Uxz)
(6)
where a denotes the sigmoid function, and Pa, (x( is the negative word sampling distribution used to draw A negative samples.
We modify skip-gram by computing numeral embeddings differently from non-numerical word em- beddings. We associate each prototype number with input embedding and output embed- ding. The input and output embeddings of all the prototypes constitute matrices Mj E Dx|P| and Mo E IR DxPP respectively. For any numeral, we can compute its input and output embeddings by taking a weighted average of the prototype input and output embeddings respectively based on Eq and use them in exactly the same way as the embeddings of non non-numerical words to compute the learning objective (E. When drawing negative samples, we first set the ratio of numerals and non-numerical words to their actual ratio in the training corpus, to guarantee a sufficient number of numeral negative samples. Then we sample numerals and non-numerical words separately from their respective distributions in the training corpus raised to the power of 3 During training, optimize the objective function Eq 6 by back- poocc--rppggttng the gradient of error to update both the embedding matrices both the non-numerical word embedding matrices E,, Eo and the prototype number embedding matrices M, Mo. this way, the embeddings of non non-numerical words and numerals are learned jointly in the same space. We show example in Figure [
4

---

Under review as a conference paper at ICLR 2020
Methods
WS353
MEN
SIM999
Non- Non-numerical Word Vocabulary
Numeral Vocabulary
SOM
64.40
71.79
36.09
GMM
64.90
71.89
36.29
SOM, GMM, D-LSTM, Fixed
NumAsTok
65.30
71.83
35.85
vIn-vocab word).
D-LSTM
63.60
71.82
34.58
wNKward
all numerals
Fixed
64.35
72.17
36.27
SG Googleeeess
70.00
74.10
44,20
(In-vocab word),
[In-vocab numerals],
GloVe Wiki-6B
52.20
73.70
37.10
NumAsTok
wNKword
UNK
Table 2:
Results word similarity tasks trained on Wiki-1B. For reference, we also show the re- sults of the official skip-gram and Glo trained larger corpora.
Table 1: Vocabularies of different methods.
4 EXPERIMENTS AND RESULTS
We evaluate our methods on four intrinsic and extrinsic tasks: word similarity, embedding numer- acy, numeral prediction, and sequence labeling. We report results of our methods based on SOM and GMM separately. We choose the hyper-parameters (e.g., the number of prototypes, GMM ini- tialization and training methods) using validation sets and report the best hyper-paramettrs for each experiment in Appendix
4.1 BASELINES
NumAsTok This baseline treats numerals and non-numerical words in the way, which is very similar to the original skip-gram. The vocabulary includes both high-frequency words and high- frequency numerals. OOV non-numerical words are replaced with symbol UNKword and OOV numerals are replaced with symbol UNK
D-LSTM Character-level RNNs are often used to encode OOV words (Graves, 2013). Here we apply LSTM (Hochreiter & Schmidhuber 1997) to the digit sequence of numeral and use the last hidden state of the LSTM as the embedding of the numeral. We use the embedding to compute the skip-gran gram objective function and propagate the gradients back to update the LSTM. The vocabulary of digits is: {0-9, :., 't', "-' 'e').
Fixed This baseline fixed embeddings for numerals with no training. We define the embedding a numeral with value as [n); 1]/Z where f is the squashing function defined in Eq 1 E R is an all-ones vector, and a constant used to keep the vector norm close those non-numerical words and is set to 2 x D by default.
We compare the vocabularies of different methods in Table I Our methods, D-LSTM, and Fixed have finite non- numerrali vocabularies but infinite numeral vocabularies. In contrast, the NumAsTokk baseline has finite numeral vocabulary and treats all the OOV numerals UNK
4.2 WORD SIMILARITY FOR NUMERRILL WORDS
To ensure that our methods can still generate high quality embeddings for non-numerical words, we evaluate our trained embeddings on classical intrinsic word similarity tasks, including WordSim- 353, (Finkelstein et al.) (001). MEN (Bruni et al.) 2014h and Simplex-999 (Hill et al. (2014). We train 300-dimensional word embeddings on the Wikipedia dump and set the context window size to 5, the number of negative samples 5, and the vocabulary size to 3 x 10S. We the evaluation tool. provided by Jastrzebski (Jastrzebski et al. 2017). Note that while the training data contains numerals, the evaluation tasks do not involve numerals and only designed evaluate quality of non-numerical word embeddings. The results are shown in Table
It be that methods achieve scores comparable to those of the baselines. The per- formance of SG trained on 100B GoogleNews is much better than all the other methods probably because of its much larger training corpus. The results show that adding our numeral embedding methods into skip-gram does not harm the quality of non-numerical word embeddings. Additional results of our methods can be found in Appendix
httpp::/ www.:ht......uuac..aat http:://ggtthbb..cccc/uuudaaa// embeddd: ngs banchmarks
5

---

Under review as a conference paper at ICLR 2020
Magnitude
Numeration
Metrics
OVA
SC
BC
AVGR
OVA
SC
BC
AVGR
SOM
67.72
71.86
99.40
15.91
3.54
62.83
100.00
28.98
GMM
57.86
58.63
100.00
1.75
4.42
65.49
100.00
25.97
NomAsTok
12.17
51.02
95.99
144.13
7.08
61.95
99.12
27.08
D-LSTM
7.26
51.79
92.83
158.82
1.77
54.87
89.38
53.55
Fixed
83.90
78.22
100.00
1.17
0.89
49.56
99.12
56.00
Table 3: Magnitude and numeration evaluation results for our methods and baselines. Accuracies of OVA, SC and BC are expressed as percentages. Lower AVGR indicates better performance. Numbers indicating top-2 performance are highlighted.
4.3 MAGNITUDE AND NUMERATION OF EMBEDDINGS
Naik et al. (2019) propose framework for evaluating the ability of numeral embeddings to capture magnitude and numeration. Given target numeral, its embedding is evaluated against a set of numerals using the OVA (One-vs-All), SC (Strict Contrastive) and BC (Broad Contrastive) tests:
OVA: The embedding vector distance between the target and its nearest neighbor on the number line should be smaller than that between the target and any other numeral in the set.
SC: The embedding vector distance between the target and its nearest neighbor on the num- ber line should be smaller than that between the target and its second nearest neighbors on the number line.
BC: The embedding vector distance between the target and its nearest neighbor on the number line should be smaller than that between the target and its furthest neighbors on the number line.
We follow the settings described by Naik al. (2019): for the magnitude evaluation, using a set of 2342 numerals that are most frequent in Wikipedia- IB, whose embeddings are well learned by all the methods; and for the numeration evaluation, words that represent numbers (e.g., 'three', 'billion') sampled from the same corpus and we measure the distance between the target numeral embedding and the word embeddings of these words. We report the accuracy of various embedding models on these three tests, along with the average rank (denoted as AVGR) of the target numeral's nearest neighbor among all the candidates based on their vector distances to the target. We use the embeddings trained on Wikipedia-IB
the tests
run the tests using 113 English
the results. The Fixed bascline has the best performance in the magnitude evaluation, which is unsurprising because the numeral embedding vector explicitly contains the (squashed) magnitude. NumAsTok performs very well in the numeration evaluation, which because the number-represenntin words used in the evaluation are high-frequency words and their embeddings are adequately trained. Except for these two special cases, our methods can seen to outperform the baselines with a large margin.
Wallace et al. (019) recently show that classic embeddings of numerals may contain magnitude information that can be extracted by neural networks. Following their methodology, we conduct two probing tests on our 2342 numerals using multi-layer perceptrons and bilinear functions and then the resulting models to predict distances between numerals in the OVA, SC, and BC tasks. The results again show the advantage of our methods over the baselines. See Appendix for details.
4.4 NUMERAL PREDICTION
To evaluate the quality of numeral embeddings, we design a new numeral prediction task: choosing the right numeral from set of candidates given the context of the numeral in sentence.
We randomly sample 2000 sentences containing numerals from a subset of Wikipedia that is not used in training, with 600 for validation and 1400 for testing. For each sentence, use the five words preceding and following the target numeral as its context. An example shown below, where the ten bold words are the context and 2.31 is the target numeral.
In Hollywond, the average household size [2.31] and the average family size 3.00.

---

Under review as a conference paper at ICLR 2020
Wikipedia-IB. dim 300
Numeracy-600l. dim 300
SA
Sis
Sa
Sus
Metries
AVGR
MdAF
MdAPE
AVGR
MdAR
MdАPF:
AVGR
Miem-
Macro-
AVGR
Micro-
Macm-
FI
F1
Fl
F1
381.41
825.79
0,9836
455.01
1184.60
0,9880
2.91
13.50
42.74
13.66
GMM
343.50
1184.85
0,9450
444.15
1081.50
0,9866
2.19
41.86
18.47
2,02
44.07
13.77
NunAsTok
600.17
1918.00
0,9965
600.2 28
$2772.50
19.07
4.21
9.74
5.47
6.16
24.28
4.88
D.LTTM
357.45
1310.65
0.9369
466.81
1080.5
0.9908
3.98
27. 98
8.80
4.49
16.49
8.42
Fixed
50371.50
42.82
672.47
50525.00
61.59
3.23
0.00
0.00
Table 4: The results of the numeral prediction tasks.
We use all the 1400 numerals in the test set as the candidates from which one has to select the right numeral for each test sentence. Given the learned word and numeral embeddings, define two score functions rank candidate numerals given the context. Following the skip-gram model, we first define the score of center numeral predicting context word Cj s(cyln) = ei T t and the
score of context word Cj predicting the center numeral n as s(nlc,) = en Vey- Our first candidate- ranking score function SA is the sum of log probabilities of center numeral n predicting each context word Cj. We use softmax here to calculate the probability.
(eglm))
(n) Σ (ogplcy/m)
Σ log
Le,EV ealcgla) Es(ssln))
scyl/n)
Σ
log ((n)
where V, is the vocabulary of non-numerical words and (n) is the normalization factor. The other aandidate-ranking score function SB is the sum of log probabilities of each context word ej predicting center numeral
(x(ley))
SR(1) = Σ Cogp(nees) 2 Σ log Ensev, emmaley
En(nle))) - Constant
(8)
3
where W, is the set of numerals in the dataset. There are few other possible score functions, but we find that they lead to results similar SA and SB.
We use three metrics to evaluate numeral prediction (Spithourakis & Riedel 2018). MdAE is the median of the absolute errors between the predicted and true numerals, MdAPE is the median of the absolute percentage errors between the predicted and true numerals, and AVGR is the average rank of the true numeral among the candidates. Detailed formulas of the three metrics are shown in Appendix E
We train embeddings on Wikipedia- IB and report the evaluation results in the left part of Table 4 Our methods significantly outperform the NumAsTok and Fixed baselines on all the three metries. D-LSTM also performs well but needs more parameters and computing time than our methods.
We also conduct slightly different numeral prediction task the recently released Numeracy- 600K dataset (the Article Title part) (Chen et al.) 2019). This dataset contains 600k sentences with numerals and in each sentence, one numeral is selected and tagged with its order of magnitude. There are eight possible orders magnitude and the goal is to predict the correct one for the target numeral from its context. To solve this multi-class classification problem, sample 100 numer- als for each order of magnitude and use the mean of their numeral embeddings to create a 'meta' embedding: we then use these 'meta" embeddings to replace the numeral embeddings in the score functions SA and Sy and the highest-scoring order of magnitude is returned.
We split the dataset to 450k sentences for training, 50k for validation and 100k for testing. We use micro-Fl and macro-Fl in addition to AVGR as the evaluation metrics. The result is shown in the right part of Table The result shows that our methods achieve much better performance compared to the baselines.
4.5 SEQUENCE LABELING ON CUSTOMER SERVICE DATA
To verify the effectiveness of our methods in practice, we evaluate our methods with a sequence labeling task dataset of customer service chat log from online apparel shopping website.
7

---

Under review as a conference paper at ICLR 2020
Original
Augmented
Hard
Acc
P
R
Acc
P
F1
Acc
P
R
FI
GMM
97.12
91.19
90.83
91.28
90.
90.72
96.19
86.66
85.91
86.28
SOM
97.04
90.74
90.45
90.60
97.03
91.19
90.43
90.81
96
86.18
85,93
86.06
100%
D-LSTM
96.72
89.84
88.80
89.32
96.72
90.40
88 88.99
89.69
95.52
84.19
83,30
83.74
Fixed
95.75
86.19
87.42
86.80
95.86
87.13
87.65
87.39
93.97
78.39
80.18
79.27
NumAsTok Tok
96.88
91.37
89.29
90.32
96. 36
90.99
87.39
89.15
96.00
87.11
85.12
86.10
GMM
96.21
89,55
86,07
87.78
95,92
89.07
85,33
87.16
95.27
84,42
81.62
82,99
SOM
96.20
89.50
86.18
87.81
95.88
89.12
85.29
87.16
95.23
84.44
81,50
82.94
30%
D-LSTM
95.55
86.83
83.88
85.33
95.30
86.22
83.13
84.64
94.32
80.10
78.17
79.12
Fixed
94.67
83,51
82.69
83,10
94,48
83.40
82.02
82.71
92.92
75.03
75,18
75.10
NumAsTTk
95.58
89.18
83.55
86,
94.57
88.39
79.94
83.95
94.65
84.42
79.06
81.65
GMM
93.43
$2.36
75.01
78.51
92.78
81.48
72.85
76.92
93.19
80.26
72.71
76.30
SOM
93.48
82.13
75.11
78.46
92.87
80.96
73.22
76.89
93.24
79.47
73.04
76.11
10%
D-LSTM
92.53
77.71
71,45
74.45
91.99
76.24
69,96
72.96
92.10
73.26
68.72
70.92
Fixed
91.90
75.39
71.41
73.34
91.48
73.96
70.20
72.02
91.06
69.50
67.47
68.46
NumAsTok
92.31
81.98
70.51
75.81
80.10
64.95
71.73
92.00
79.64
67.95
73.32
Table 5: The results of sequence labeling. We report the accuracy, precision, recall, F1 score for the original, augmented, and harder test sets with different training data sizes. Accuracy is in the token level and the other metrics are in the entity level. We report the standard deviations in Appendix H
This dataset contains large number of numerals related to height, weight, foot length, etc., and therefore is good testbed for evaluating numeral embeddings.
The task is to assign label to each word numeral in the dataset indicating its information type. We shows two examples below:
W O H O
O
W
H 0 0
82 kg 177 what size shall choose
82 177 what size
W. H. O are labels representing weight, height and ordinary word respectively. We show the statistics of the dataset in Appendix In order better evaluate the generalizability, we create two additional test sets. The first one is created by 'augmenting' the original test set with new sentences containing slightly perturbed numerals. For example, we can create new sentences by replacing '177' in the above example with '176' and '178'. The second one contains 'hard' sentences from the original test set that do not have explicit cues for label prediction. For example, the first sentence above contains 'kg' and 'cm' that can greatly facilitate the prediction of W and H, but the second sentence above does not contain such cues and hence 'hard' sentence. More details about the two test sets can be found in Appendix F Finally, we also test the low-resource settings in which only 30% or 10% of the training set is used.
We learn embeddings from the training set using our methods and the baselines and use a validation set to do model selection. We plug the learned embeddings into the Neural-CRF model (Yang & Zhang 2018) to do sequence labeling without using part-of-speech and character-level features and embedding fine-tuning.
The results shown in Table 5 Our methods consistently outperform all the baselines the Ac- curacy, Recall, and Fl metrics in different configurations. NumAsTok trained with 100% training samples has the highest precision the original and hard test sets probably because it learns high- quality embeddings for high-frequency numerals included in its vocabulary; but its recall is lower than that of our methods, most likely because of its numeral OOV problem. Comparing the results on the original and augmented test sets, we see that NumAs Asoo shows a more significant drop in performance than the other methods, which suggests that NumAsTok does not generalize well be- cause of the numeral OOV problem. In the low-resource settings, the advantage of our methods over the baselines becomes even larger, indicating better generalizability and less annotation required for our methods to achieve a promising performance.
5 CONCLUSION
In this paper, we propose two novel numeral embedding methods that represent the embedding of a numeral as a weighted average of a set of prototype numeral embeddings. The methods can be
https://githube http::///ggihhb..ooooieeeeeee/ee htte:://ggthbbb.cco///eettttt/// htwp://t/ocm/eette/////cce
8

---

Under review as a conference paper at ICLR 2020
integrated into traditional word embedding approaches such as skip-gram for training. We evaluate our methods on four intrinsic and extrinsic tasks, including word similarity, embedding numeracy, numeral prediction, and sequence labeling, and show that methods can improve the performance of numeral-related tasks and has better generalizability. Our code and sample data can be found at path/to. code/
An important future direction is to handle numeral polysemy. For example, the numeral "2019" may denote either year an ordinary number. One potential method is assign a different embedding to each sense of a numeral. In this way, "2019" would have one embedding for representing year and another for representing an ordinary quantity. The similarity function would treat different senses of numeral differently. For example, the year sense of "2019" would be similar to the year sense of "19" but dissimilar to the sole sense of "2019.5", while the quantity sense of "2019" would be similar to that of "2019.5".
REFERENCES
Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Jauvin. A neural probabilistic language model. Journal of machine learning research, 3(Fcb): 2003.
Johannes Blömer and Kathrin Bujna. Simple methods for initializing the algorithm for gaussian mixture models. CoRR, 2013.
Elia Bruni, Nam Khanh Tran, and Marco Baroni. Multimodal distributional semantics. Artif. Int. Res., 49(1):1447, Jamuary 2014. ISSN 1076-9757.
cfm?id=2655713 655714
John Bullinaria and Joseph P Levy. Extracting semantic representations from word co-occurrence statistics: A computational study. Behavior research methods, 39(3): 2007.
Chung- Chi Chen, Hen-Hsen Huang, Hiroya Takamura, and Hsin-Hsi Chen. Numeracy-6000 Learn- ing numeracy for detecting exaggerated information in market comments. In Proceedings of the 57th Conference of the Association for Computational Linguistics, pp. 6307-6313, 2019.
Ronan Collobert and Jason Weston. A unified architecture for natural language processing: Deep neural networks with multitask learning. In Proceedings of the 25th international conference on Machine learning, pp. 160-167. ACM, 2008.
Marie-Catherine De Marneffe, Anna N Rafferty, and Christopher D Manning. Finding contradic- tions in text. Proceedings of ACL-08: HLT, pp. 1039-1047, 2008.
Stanislas Dehaene. The number sense: How the mind creates mathematics. OUP USA, 2011.
Stanislas Dehaene, Manuela Piazza, Philippe Pinel, and Laurent Cohen. Three parietal circuits for number processing. Cognitive neuropsychology. 20(3-6):487-506, 2003.
Lev Finkelstein, Evgeniy Gabrilovich, Yossi Matias, Ehud Rivlin, Zach Solan, Gadi Wolfman, and Eytan Ruppin. Placing search in context: The concept revisited. In Proceedings of the 10th International Conference on World Wide Web, www '01, pp. 406-414, New York, NY, USA, 2001. ACM. ISBN 1-58113-348-0.
doi: 10.1 10.11455371920.372094
.ccm
oxc//00.144/571120..33 372094
Alex Graves. Generating sequences with recurrent neural networks. Computer Science, 2013.
Felix Hill, Roi Reichart, and Anna Korhonen. Simlex-999: Evaluating semantic models with (gen- uine) similarity estimation. CoRR, abs/1408..446, 2014. URL http://arxiv..rgg// /bs/
7408.3457
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Ander Intxaurrondo, Eneko Agirre, Oier Lopez De Lacalle, and Mihai Surdeanu. Diamonds in the rough: Event extraction from imperfect microblog data. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 641-650, 2015.

---

Under review as a conference paper at ICLR 2020
Stanislaw Jastrzebski, Damian Lesniak, and Wojeiech Marian Czarnecki. How to evaluate word em- beddings? on importance of data efficiency and simple supervised tasks. CoRR, abs/1702.02170, 2017. URLptttp:/aaxxi..oo org a22. 02177022221000
Teuvo Kohonen. The self-organizing map. Proceedings of the IEEE, 7809):146-11483 1990.
Rémi Lebret and Ronan Lebret. Word emdeddings through hellinger PCA. CoRR, abs/1312.5542, 2013. /ttp:/tee///rrai...or///////////
Iddo Lev, Bill MacCartney, Christopher Manning, and Roger Levy. Solving logic puzzles: From robust processing to precise semantics. In Proceedings of the 2nd Workshop on Text Meaning and Interpretation, 2004.
Kevin Lund and Curt Burgess. Producing high-dimennional semantic spaces from lexical co- occurrence. Behavior research methods, instruments, & computers, 28(2):203-088, 1996.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9Nov))2579-22605 2008
Aman Madaan, Ashish Mittal, Ganesh Ramakrishnan, Sunita Sarawagi, et al. Numerical relation extraction with minimal supervision. In Thirtieth AAAI Conference on Artificial Intelligence, 2016.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word represen- tations in vector space. Proceedings of the International Conference on Learning Representations (ICLR 2013), 2013a.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representa- tions of words and phrases and their compositionality. In C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q Weinberger (eds.), Advances in Neural Information Processing Systems 26, pp. 3111-3119. Curran Associates, Inc., 2013b.
Arindam Mitra and Chitta Baral. Learning to use formulas to solve simple arithmetic problems. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume I, pp. 2144-2153, 2016
Aakanksha Naik, Abhilasha Ravichander, Carolyn Rose, and Eduard Hovy. Exploring numeracy in word embeddings. In Proceedings of the 57th Annual Meeting of the Association for Computa- tional Linguistics, pp. 3374-3380, 2019.
Andreas Nieder and Earl Miller. Coding of cognitive magnitude: Compressed scaling of numeri- cal information in the primate prefrontal cortex. Neuron, 37(1): 2003.
Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word
representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 1532 1543, 2014.
Douglas LT Rohde, Laura M Gonnerman, and David C Plaut. similarity based lexical co-oceurrence. Communications of the ACM, 8(627-633):166, 2006.
An improved model of semantic
Subhro Roy and Dan Roth. Solving general arithmetic word problems. 1rXivv1608.01411 2016.
arXiv preprint
Subhro Roy, Tim Vieira, and Dan Roth. Reasoning about quantities in natural language. Transac- fions of the Association for Computational Linguisties, 3:1-13, 2015.
Georgios Spithourakis and Sebastian Riedel. Numeracy for language models: Evaluating and im- proving their ability to predict numbers. In Proceedings of the 56th Annual Meeting of the As- sociation for Computational Linguistics (Volume I: Long Pupers), pp. 2104-2115, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10.18653/v//P1. 1196. htt.tttss::/ www. ac 1 web. org/ httg://nhoolooyyy/o 8-111-119966
Georgios P. Spithourakis, Isabelle Augenstein, and Sebastian Riedel. Numerically grounded lan- guage models for semantic error correction. CoRR, abs/1608.04447, 2016. URL http:
7/arxiv. //axxiv..rrg/ab//1100 04147
10

---

Under review as a conference paper at ICLR 2020
Shyam Upadhyay, Ming-Wa Wei Chang, Kei Chang. and Wen-tau Yih. Learning from explicit and implicit supervision jointly for algebra word problems. In Proceedings of the 2016 Conference Empirical Methods in Natural Language Processing, pp. 297-306, 2016.
Eric Wallace, Yizhong Wang, Sujian Li, Sameer Singh, and Matt Gardner. Do nlp models know numbers? probing numeracy in embeddings. 2019.
Yan Wang, Xiaojiang Liu, and Shuming Shi. Deep neural solver for math word problems. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 845-854, Copenhagen, Denmark, September 2017. Association for Computational Lin- guisties. doi: 10.18653/1/D117-0088. URL https: //tp:://wwwww/ wwwwwb.wwwbcc..... org www.ortttggogyy///ooooooooooo.ot D17-1088
Jie Yang and Yue Zhang. Nerf++:
An open-source neural sequence labeling toolkit. In Proceedings of the 56th Anmual Meeting of the Association for Computational Linguistics, 2018. URL URLhttp:
A
GMM INITIALIZATION
Both EM and hard-EM are sensitive to initialization and we use the initialization methods described in (Blömer & Bujna 2013). We first initialize the mean μl of the k-th Gaussian component using one of the following three strategies:
Random initialization: choose μ* from X randomly. This is suitable when X contains wide range of numbers, e.g., numbers collected from Wikipedia.
SOM-based initialization: initialize μ4 pk IP produced by the SOM method.
K-means initialization: run randomly initialized k-means X and then use k-means centroids to initialize μik-
We then assign the data samples to their closest means. The standard deviation of the data samples assigned to the k-th mean becomes og.
B
We list all of the important hyper-parameters we tune for each model.
General Syperrparammetes: embedding dimension, context window size, SGD learning rate, batch size, vocabulary size, etc.
SOM
number of prototypes, stage of applying the log-squashing function (stage I: before prototype induction; stage only the similarity function).
GMM hyper-parameters number of prototypes, whether we apply the log-squashing function to the numerals, EM initialization (from SOM, random initialization, or k-means initialization), type of EM (hard-EM or soft-EM).
We show the values of the SOM and GMM hyper-parameters in Table the values of the general of all the methods in Table We find that the general hyper-parameters influence the performance of our methods and the baselines in the same way, so most cases, these hyper- parameters are set to be identical for all the methods. For large training corpora (WikilB, Numeracy- 600k), we use 2048 as the batch size for D-LSTM, because D-LSTMM consumes much more GPU memory. We set the batch size of the other methods to 4096. For the sequence labeling tasks, because the data relatively small and confined to a very specific domain (chat log from online apparel shops), we set small vocabulary size of 500 for all the methods except NumAsTok and set the vocabulary size of NumAsTok to 550 to ensure that different methods have similar numbers of parameters for word embedding training. Consequently, our methods have (500 + PPI) D parameters for word embedding training and NumAsTok has 550 x parameters, where the prototype set, whose size is typically smaller than 50, and D is the embedding dimension.

---

Under review as a conference paper at ICLR 2020
T shows that the optimal number of prototypes is around 500-500mm for the WikilB corpus and 10-25 for the much smaller sequence labeling dataset. As a rule of thumb, we suggest setting the number of prototypes to (log 2 where N is the number of distinct numerals in the training
corpus.
SOM
GMM
prototype number
log trunsform
protocype number
transfor
initialization
Word similarity (WIG1B)
200
dataset
200
True
random
hard
Magnitude (WkiiB)
200
dataret
300
True
random
soft
Numeration (Wikil
similarity function
random
Numeral Prodiction (WikilB)
similarity function
300
False
random
hard
Numeral Prediction (Numeracy- 600k)
50
datoset
200
False
random
hard
Sequence Labeling 100 f
dataset
30
False
random
soft
Sequence Labeling
False
Soquence Labeling 10% %
25
similarity Function
20
False
from-som
solt
Table 6: Hyper-parameter values for GMM and SOM based methods for each experiment.
embedding
context window
negative samples
epoch
batch size
learning rate
vocabulary
dim
size
Word similarity (WikiTB)
300
5
4096, 2048
5 10
3 10"
Magnitude (-MAG) (WikilB)
300
4096, 2048
5
3
Numeration (-NUM) (Wiki1B)
300
4096, 2048
10-3
106
Numeral Prediction (WikiBB
300
4096, 2048
5 10
3 10
Numeral Prediction (Numeracy-6ule
300
10
4096, 2048
5 10
x 10
Sequence Laheling 100% 30% 10%
5.x 10-2
500, 550
Table 7: Values of general hyper-paramettn for each experiment.
C MORE RESULTS ON WIKIPEDIA-IB 1B
167
201
207
numbro
10l
100
10 36
10f
1014
10
367
106
nimaral
(a) Numerals in Wikipedia IB
(b) Prototypes of SOM-500
(c) Prototypes of GMM-500-soft
Histograms of numerals and learned prototypes that range from 0 to 10 The horizontal axis represents the numeral quantity and the vertical axis represents the number of occurrences, 5500' means the number of prototypes, 'soft' means soft-EM.
Figure 2:
We show the histograms of numerals in the Wikipedia- 1B dataset and the prototypes learned by SOM and GMM in Fig[2] can seen that the prototypes induced by our methods have a similar distribution compared to the original numerals.
We also show some examples prototypes and their nearest non-numerical words in Table g We use the embedding trained by the SOM model with 200 prototypes on Wikipedia-1B. and use log transformation in the similarity function.
In addition, select several typical numerals and non-numerical words and project their embed- dings to 2D using t-SNE (Maaten & Hinton 2008) (Figure 3). We use embeddings learned on Wikipedia-1B corpus using the SOM and GMM methods. The examples and the figures show that our model does capture some semantic relations between numeral quantities and normal words.
We show the training speed of each embedding method the Wikipedia- 1B dataset in Table 9 The batch size is set to 2048 for all the methods. Our methods are slower than NumAs Tok but are faster than D-L.TTM.
12

---

Under review as a conference paper at ICLR 2020
Prototype Value
Most Related Non-numerical Words
8 8186446.58
million, billion, total, budget, funding, estimated, dollars thousand. approximately, thousands, millions. roughly, hundreds nearly, millennium, decade, internet, twentieth, worldwide, latest johann, renaissance, giovanni, dutch, baroque, vii, shakespeare ten, six, eleven, pm, seconds, eight
10372.49
2000.06
1,59.799
10.00
Table 8: Examples of prototypes and their nearest non-numerical words.
Method
SOM
GMM
NumAsTok
D-LSTM
Fixed
Speed (sent/s)
13590.93
12691.18
22907.97
8421.66
13055.08
Table 9; Training speed for each methods.
D PROBING TESTS
We apply two probing tests using neural network on our methods and baselines in order to compare their ability to encode magnitude information in a non-linear way. The first test Decoding (pre- dicting the numeral value from its embedding using MLP). The second is Subtraction (predicting the difference between two numerals from their embeddings using MLP or BiL.inaar functions). We illustrate the tasks and the models we use in Figure
We first create the datasets for the two probing tests based on the dataset from the magnitude eval- uation of Section 4.3 (containing 2342 numerals). For Decoding, the dataset can be directly used. For Subtraction, we randomly sample 10$ pairs numerals (Mi (22) from the dataset and assign randomly split 80% of each
71 n2 the prediction target. Following Wallace et al. (2019).
dataset for training and 20% for testing. We use SGD to optimize the mean square error (MSE) loss.
We report the root-mean-squan error (RMSE) results for the two tasks in Table T0
Decoding
Subtraction
MLP1
MLP2
BiLinear
BiLinear+MLPI
MLP1
MLP2
GMM
77.73
68.29
1006.66
62.04
18.10
4.40
SOM
289.40
182.91
2109.16
100.03
33.62
14.04
NumAsTok
1027.33
1035.25
2835.32
282.32
15.52
4.66
D-LSTM
1433.45
1397.00
3208.96
634.20
717.04
644.41
Fixed
2220.75
2225.31
3791.96
3790.93
3791.91
3791.98
Table 10: Probing test results. respectively.
MLP1 and MLP2 denote MLP with one and two hidden layers
The results show that our two methods are significantly better than the baselines on Decoding. On Subtraction, they are better than the baselines when using BiLinear and are comparable to NumAs- Tok but much better than the other baselines when using MLP. We found that the performance is very sensitive to the neural network architecture and MLP with two hidden layers performs best.
We then use the MLP2 models trained on Subtraction to determine the distance between two nu- merals when conducting the magnitude evaluation Section 4.3 The results are shown in Table
The results show that our methods have better performance than the baselines overall. One interesting observation is that, although our SOM based method has worse RMSE than NumAsTok as shown in Table 10 it outperforms NumAs in the magnitude evaluation.
E NUMERAL PREDICTION EVALUATION METRICS
We denote the target numeral by n, the numeral with the highest ranking score by ân and the rank of the target numeral by r;. The error e; and percentage error pe; be calculated as:
G n4
ń,
pe
(9)
n
13

---

Under review as a conference paper at ICLR 2020
100
population
million
1000000.0
thousand
1000000000.0
twenty
9.5
eleven
211.3
year
11.0
november
october
2000.00 millennium
-50
1599.3
renaissance
shakespeare
-100
-50
50
100
(a) (-SNE plot embedding trained by the SOM- method with 200 prototypes.
150
211.3
100
shakespeare
11.0
renaissance
october
2000.001
millennium
year
november
1599.3
eleven
-50
twenty
million
1000000000.0
cm
thousand
1000000.0
-100
-150
population
-150
-100
-50
100
200
(b) 1-SNE plot for embedding trained by the GMM- method with 300 prototypes, random initialization and soft-EM training.
Figure 3: t-SNE results for the SOM-based and GMM-based methods.
Then we use the median of the absolute errors, the median of the absolute percentage errors, and the average rank as the evaluation metries.
MdAF (eel),
MdAPE mepean
AVGR
(10)
14

---

Under review as a conference paper at ICLR 2020
Decoding (Regression)
Subtraction
(Begiession)
(Regression)
MGB
MSE
7.1
19
17.3
8
7.1
0000
MLLP
HLP
BiLinear
0
O
10000
oooo
0000
Pre-treined Rubaddar
PrR-trained Emsedder
Pre-treined
Embedder
Figure 4: Diagrams of the probing models for Decoding and Subtraction.
OVA
SC
BC
AVGR
GMM
8.03
48.76
100.00
16.45
SOM
4.40
57.43
100.00
59.41
NumAs'
1.96
50.90
100.00
106.98
D-LSTM
0.51
52.09
100.00
545.87
Fixed
0.00
00.00
0.00
1170.68
Table 11: Results of the magnitude evaluation using MLP2 trained on the Subtraction probing task. Accuracies of OVA, SC and BC are expressed percentages, Lower AVGR indicates better perfor- mance.
F AUGMENTED AND HARD TEST SETS IN SEQUENCE LABELING
The augmented test set created by reasonably perturbing the numerals sentence. For example, for a numeral '173' that describes height, we generate new samples by changing '173' to '174' or '175' while keeping the other non-numerical words in the sentence unchanged. For decimal such as *1.7 meters', we change it to '1.6' or *1.8'. The perturbation will not change the decimal places of numerals and will only change the quantity slightly, which makes the generated sentences reasonable.
The hard test set created by manually collect 'hard' samples the original test set. Hard samples do not have explicit patterns, meaning that a numeral's 's tag cannot be easily inferred by its adjacent words. For example, tags numerals followed by units like 'cm'. 'm', 'kg'. 'years' and 'feet' can be figured out easily,
exclude them from the hard test set. Customers are very likely to use ambiguous expressions like: 'I'm 16.5, can buy 24?', where 16.5 about foot length and 24 the shoe size. These ambiguous sentences included in the hard test set.
G STATISTICS OF SEQUENCE LABELING DATASET
The vocabulary is small because the dataset is confined to a specific domain: online customer service chat log about apparel purchase. In this dataset, most of the sentences are about sizes of various kinds of clothes and very short and ambiguous.
We show the statistics of the customer-service dataset in the Table 12
H SEQUENCE LABELING RESULT WITH STANDARD DEVIATION.
15

---

Under review as a conference paper at ICLR 2020
Number of Sentences
Train 1389
Dev 793
Original Test 1802
Augmented Test 8052
Hard Test 726
Statistics of Training Set
Token Vocab 505
Numeral Vocab
Avg sent length
Numeral Ratio
labels 21
234
10.42
15.89 %
Table 12: Statistics of low-resource customer-service dataset.
Original
Augmented
Hard
Acc
P
F1
97.12
91.19
90.46
90.83
97.02
91.28
90.18
90.72
96.19
86.66
85.91
86,28
+0005
40.11
+00.7
400.4
+0.06
+-0.17
+00.99
4-0.18
+00.5
+-0.13
-0.36
+-0.24
90.60
97.03
91.19
90.43
90.81
96.06
100%
SOM
+0.04
+0.15
+00.10
+00.1
+0.03
+00.14
+0.11
+00.33
+00.9
+00.14
-1132
+-0.23
96.72
89.84
89.32
96.72
90.40
88.99
89.69
95.52
84.19
83,30
83.74
D-LSTM
+0006
+60.26
0.2.4
10.255
#0007
++0.29
0.23
10.26
+0.14
+:0.66
0..55
+-0.60
95.75
86. 19
87.42
86.80
95.86
87.
87.65
87.39
93.97
78.39
80.18
79.27
Fixed
::0.22
1:0.26
0.43
96.88
91.37
89,29
90,32
96.36
90.99
87.39
89.15
96.00
87.11
85,12
86.10
NumAsTok
-0.07
4:0.40
+.0.99
+-0.21
0.05
+0.41
0.09
+0.20
=00
+:0.52
-0.03
+-0.26
GMM
=0.07
+50.15
+-032
+-0.24
00009
+-0.22
0000
+0.36
-3.13
+10.41
00.47
+-0.33
96.20
89.50
86.18
87.81
95.88
89.12
85.29
87.16
95.23
84.44
81.50
82.94
30%
SOM
+:0.33
+-0.77
+0229
++0.88
10.08
+60.16
-0.41
+0.13
+00.2
+-0.30
0,22
+-0.09
95.55
86.83
83.88
85.33
95.30
86.22
83.13
84.64
94.32
80.
78.17
79.12
D-LSTM
+0.50
94.67
83.51
82.69
83.10
94.48
83.40
82.02
82.71
92.92
75.03
75.18
75.10
20.66
100.1
10.18
+00.12
0.08
+0.23
10226
10.17
20.05
1:0.06
=0.38
+0.17
88.39
79.94
83.95
NumAsTok
-0.03
+00.55
#:0.3
+00.10
0.00
+:040
10.16
+0.21
20.03
+:0.39
=0.23
+00100
93.43
82.36
75.01
78.51
92.78
$1.48
72.85
76.92
93.19
80.26
72.71
76.30
GMM
-2.12
:10.17
+0052
+0021
+50.25
00.36
+0.14
=0.04
+10.41
+0.199
93.48
82.13
75.11
78.46
92.87
80.96
73.22
76.89
93.24
79.47
73.04
76.11
10%
SOM
10.11
:10.26
10.41
100.1
00.10
1:0.25
10.37
10.19
10.10
1:0.07
20.49
1:030
92.53
77.71
71.45
74.45
91.99
76.24
69.96
72.96
92.10
73.26
68.72
70.92
D-LSTM
10019
10.38
+0.82
+0.61
20.26
10.40
10.11
10.80
10.16
1:0.66
=0.70
10.40
Fixed
72.02
-0.05
+0.46
+0.58
+:0.17
20.12
10.64
-0.73
10.40
10.66
1-0.78
=0.27
+0025
92.31
81.98
70.51
75.81
90.77
80.10
64.95
71.75
92.00
79.64
67.95
73.32
NumAsTok
20.12
10.44
10.56
10.65
10.66
40.23
10.06
10.64
=0.38
10.20
Table 13: The results of sequence labeling. We report the accuracy, precision, recall, Fl score for the original, augmented, and harder test sets with different training data sizes. Accuracy is in the token level and the other metrics are in the entity level.
16


Reviews:
[{"review_id": "Hke6V2XEFB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nSummary:\nThe paper talks about a recently highlighted problem in word embeddings which is their incapability to represent numerals, especially the out-of-vocabulary numerals. For addressing the problem, they propose a method that induces a finite set of prototype numerals using either self-organizing map or Gaussian Mixture model. Then, each numeral is represented as a weighted average of prototype numeral embeddings. The method also involves squashing large quantities using log function. Finally, the training is performed similar to Skip-gram in word2vec but with the embedding of numerals computed using prototype numerals. \n\nQuestions:\n1. There are two basic motivations of the paper: (1) Most of the word embedding methods do not embed numerals correctly. (2) There is no mechanism of handing OOV numerals. The second one is well addressed but for the former one, it has been shown in recent work [1] that most of the embedding methods do have numerical reasoning capabilities. As stated in [1], the results of [2] demonstrate the opposite conclusion because their analysis is based on cosine distance and nearest neighbor which are not capable of capturing non-linear dependencies between embeddings. \n\nSo, it would be great if, for the results in Section 4.3 instead of using cosine distance, some neural models could be utilized for evaluation (3 layer MLP similar to [1]).\n\n2. In Section 4.4, the task is to predict the target numeral given its context words (similar to CBOW) while the embeddings are trained with a modified skip-gram model. Can one expect superior results if one uses modified CBOW for training embeddings rather than skip-gram?\n\n3. In Table 5, with 100% training data, the performance of all the methods is very close. It would be better if mean and variance across multiple runs are reported. ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "B1lamhMaYr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I have read the author response.  Thank you for responding to my concerns.\n\nOriginal review:\nThis paper presents a word embedding approach for numbers.  The method is based on finding prototype numbers, and then representing numbers as a weighted average of the prototype embeddings, where the weights are based on numeric proximity.  The approach provides some gains over baselines on number similarity, number prediction, and sequence tagging tasks.  While modeling numbers is an interesting task, several aspects of the paper needed more clarification, and the paper’s focus may be somewhat narrow for the ICLR audience.\n\nI think the paper could use a better motivation for the prototype-based approach.  In particular, the fact that the approach uses only quantity (and not the form of numbers) to represent similarity is contrary to my intuition.  For example, I would expect 1960 and 1960.1 to behave very differently in text, because one is a year and the other isn’t.  But the proposed method, if I understand it correctly, would give them similar embeddings because they are very close numerically.\n\nI was not able to understand the SOM portion of the method, it is not self-contained within this paper.\n\nOn the Numeracy-600K data set, Chen et al. (2019) shows much higher F1 results than those shown here.  What explains this difference?\n\nThe proposed method seems relatively similar to that of (Spithourakis & Riedel, 2018), in that it exposes numeric quantity to the language/embedding model, and uses GMMs to represent numeral distributions.  More clarity about how this approach compares to that one (and others) would be helpful.  Also, how does the (Spithourakis & Riedel, 2018) approach fare on e.g. the number prediction tasks in the submission?  It is true that the submission's approach produces general-purpose embeddings that can be re-used, unlike (Spithourakis & Riedel, 2018).  But we would like to know whether that generality comes at the cost of performance on tasks, and if so how much of a cost.\n\nMinor:\nThe Lund and Burgess reference seems incorrect.  I don’t see that those authors published a paper by that title in Brain and Cognition in 1996.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "B1lHzT5atH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nThe paper proposes a novel method for embedding numerals which can be learned by using neural word embedding learning techniques. The paper motivates the work by reviewing the difficulty of embedding components to represent numerals: OOV in most cases. Their main contribution is the introduction of a method composes numeral embedding by a weighted average of prototype embeddings based on the similarities between the numeral and prototypes. There are two proposed prototypes: SOM and GMM and the similarity functions are an absolute difference and the density function respectively. During the training, the numerals have the proposed embeddings while the others have normal word embeddings. The paper slightly modifies the negative sampling to ensure numerals being sampled. A series of 4 empirical studies have been presented. First, the paper confirms that the proposed method does not negatively affect non-numeral embeddings. And then, the quality of the numeral embeddings are evaluated and compared. The experiments show that the proposed method has better performance on numerical property tests, numeral prediction, and a sequence labeling task.\n\nOverall, this paper has a novel contribution. The proposed method is well motivated and quite justified by the experiments abliet lacking comparison with previous published results. However, it has some weaknesses.\n\nFor the method part, one of the limitations is that it is not an end-to-end method and requires a regular expression to identify the numeral. Second, the weighted average of the prototypes is reasonable, but the similarity function only relies on the magnitude. I think there are other aspects of numerical tokens that it might not be able to capture (e.g. “2019” is similar to “19” in some context). In terms of training, I think it is not hard to extend the method to full language model training (using softmax). However, adding all numerals to the vocabulary would add significant overhead.\n\nFor the experiments, some design decisions are left unjustified. For example, a simple ablation experiment on the squashing function will be helpful. Furthermore, guidelines or empirical results on the effect of the number of prototypes can increase the impact of the paper. Finally, I think an analysis of the performance of numerical types will be helpful for future works (e.g., dates, phone numbers, currencies, etc, or discrete vs continuous). \n\nMinor comments and questions:\n1. The log sigmoid in equation 6 is a bit strange, isn’t it log sum exp(). https://arxiv.org/pdf/1402.3722v1.pdf\n2. Why do you create a new dataset for the experiment in section 4.3?\n3. In section 4.4, you rank only numerals in the test set, but the scores are computed based on all numerals in the vocab. Do you have the performance of ranking all numerals?\n4. Just to confirm the “training” in section 4.4 refers to learning the embedding using the skip-gram model, right?\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes better methods to handle numerals within word embeddings.

Overall, my impression is that this paper is solid, but not super-exciting. The scope is a little bit limited (to only numbers), and it is not by any means the first paper to handle understanding numbers within word embeddings. A more thorough theoretical and empirical comparison to other methods, e.g. Spithourakis & Riedel (2018) and Chen et al. (2019), could bring the paper a long way.

I think this paper is somewhat borderline, but am recommending not to accept because I feel that the paper could be greatly improved by making the above-mentioned comparisons more complete, and thus this could find a better place as a better paper in a new venue.
Decision: Reject