Title: LEARNING NUMERAL EMBEDDING

Abstract: Word embedding is an essential building block for deep learning methods for natural language processing. Although word embedding has been extensively studied over the years, the problem of how to effectively embed numerals, a special subset of words, is still underexplored. Existing word embedding methods do not learn numeral embeddings well because there are an infinite number of numerals and their individual appearances in training corpora are highly scarce. In this paper, we propose two novel numeral embedding methods that can handle the out-ofvocabulary (OOV) problem for numerals. We first induce a finite set of prototype numerals using either a self-organizing map or a Gaussian mixture model. We then represent the embedding of a numeral as a weighted average of the prototype number embeddings. Numeral embeddings represented in this manner can be plugged into existing word embedding learning approaches such as skip-gram for training. We evaluated our methods and showed its effectiveness on four intrinsic and extrinsic tasks: word similarity, embedding numeracy, numeral prediction, and sequence labeling.

1. INTRODUCTION:

Word embeddings, the distributed vector representations of words, have become the essential building block for deep learning approaches to natural language processing (NLP). The quality of pretrained word embeddings has been shown to significantly impact the performance of neural approaches to a variety of NLP tasks. Over the past two decades, significant progress has been made in the development of word embedding techniques (Lund & Burgess, 1996; Bengio et al., 2003; Bullinaria & Levy, 2007; Mikolov et al., 2013b; Pennington et al., 2014) . However, existing word embedding methods do not handle numerals adequately and cannot directly encode the numeracy and magnitude of a numeral (Naik et al., 2019) . Most methods have a limited vocabulary size and therefore can only represent a small subset of the infinite number of numerals. Furthermore, most numerals have very scarce appearances in training corpora and therefore are more likely to be outof-vocabulary (OOV) compared to non-numerical words. For example, numerals account for 6.15% of all unique tokens in English Wikipedia, but in GloVe Pennington et al. (2014) which is partially trained on Wikipedia, only 3.79% of its vocabulary is numerals. Previous work (Spithourakis et al., 2016 ) also shows that the numeral OOV problem is even more severe when learning word embeddings from corpora with abundant numerals such as clinical reports. Even if a numeral is included in the vocabulary, its scarcity in the training corpus would negatively impact the learning accuracy of its embedding.

The inadequate handling of numerals in existing word embedding methods can be problematic in scenarios where numerals convey critical information. Take the following sentences for example, "Jeff is 190, so he should wear size XXL" (190 is a reasonable height for size XXL) "Jeff is 160, so he should wear size XXL" (160 is an unreasonable height for size XXL) "Jeff is 10, so he should wear size XS" (10 is an age instead of a height) If the numerals in the example are OOV or their embeddings are not accurately learned, then it becomes impossible to judge the categories of the numerals or the reasonableness of the sentences.

In this paper, we propose two novel methods that can produce reasonable embeddings for any numerals. The key idea is to represent the embedding of a numeral as a weighted average of a small set of prototype number embeddings. The prototype numerals are induced from the training corpus using either a self-organizing map (Kohonen, 1990) or a Gaussian mixture model. The weights are computed based on the differences between the target numeral and the prototype numerals, reflecting the inductive bias that numerals with similar quantities are likely to convey similar semantic information and thus should have similar embeddings. Numeral embeddings represented in this manner can then be plugged into a traditional word embedding method for training. We empirically evaluate our methods on four tasks: word similarity, embedding numeracy, numeral prediction, and sequence labeling. The results show that our methods can produce high-quality embeddings for both numerals and non-numerical words and improve the performance of downstream tasks.

2. RELATED WORK:

Word Embedding Word embeddings are vector representations of words which carry semantic meanings implicitly and are trained without supervision. Most existing word embedding training methods can be divided into two classes. The first class of methods (Lund & Burgess, 1996; Rohde et al., 2006; Bullinaria & Levy, 2007; Lebret & Lebret, 2013) extract word co-occurrence statistics from the training corpus, compute a word-word matrix based on measures such as PPMI, and then apply dimension reduction techniques such as principle component analysis to produce a low-dimensional vector representation for each word. The second class of methods (Bengio et al., 2003; Collobert & Weston, 2008; Mikolov et al., 2013a; b) use a simple neural network to model the relation between a word and its context within a sliding window in the training corpus. GloVe (Pennington et al., 2014) has been proposed as a method that combines the advantages of both classes. All the above methods have a finite vocabulary size and use a 'UNK' symbol to represent OOV words. Recent work (Naik et al., 2019) shows that these popular methods do not handle numerals adequately. Wallace et al. (2019) shows that existing word embedding methods can encode numeracy implicitly for high-frequency numerals, but the embedding's numeracy for OOV numerals is not investigated. Our goal is to design better numeral embedding methods that can be integrated into traditional word embedding methods and handle the OOV problem for numerals.

Numeracy in natural language Numeral understanding has been found important in textual entailment (Lev et al., 2004; De Marneffe et al., 2008; Roy et al., 2015) and information extraction (Intxaurrondo et al., 2015; Madaan et al., 2016) , but existing systems often use manually defined task-specific features and logic rules to identify numerals, which is hard to generalize to other tasks. A lot of research has been done trying to solve math problems, using either manually designed features and rules (Roy et al., 2015; Mitra & Baral, 2016; Roy & Roth, 2016; Upadhyay et al., 2016) or sequence-to-sequence neural networks Wang et al. (2017) , but the quantity of numerals is not important in this task and hence existing methods often replace numerals by dummy symbols such as n 1 and n 2 . Spithourakis & Riedel (2018) studied different strategies to better model numerals in language models. Chen et al. (2019) created Numeracy-600K dataset and studied the ability of neural network models to learn numeracy. Our work differs from previous work in that we aim to produce general-purpose numeral embeddings that can be employed in any neural NLP approach.

3. METHODS:

Given a training corpus C, we first extract all the numerals using regular expressions and form a dataset X containing all the numbers represented by these numerals. A number (e.g., 2000) may appear for multiple times in X if its corresponding numerals (e.g., '2000', '2,000', etc.) appear for multiple times in C. We then induce a finite set P of typical numerals (i.e., prototypes) from X using a self-organizing map (Kohonen, 1990) or a Gaussian mixture model. We also define a function sim(n 1 , n 2 ) outputting the similarity between two arbitrary numbers n 1 and n 2 . Now we represent the embedding of any target numeral n as a weighted average of the prototype number embeddings with the weights computed by the similarity function:

e(n) = • p∈P α • sim(n, p) • e(p), p∈P α • sim(n, p) = 1 (1)

We use e(•) to denote the embedding of a number α is the normalization factor. This formulation satisfies the intuition that numerals with similar quantities are likely to convey similar semantic information and thus should have similar embeddings.

Our numeral embeddings can be integrated into traditional word embedding methods such as skipgram for training. During training, we back-propagate the error gradient to update the prototype number embeddings. In this way, the prototype number embeddings (and hence all the numeral embeddings) are learned jointly with non-numerical word embeddings.

3.1. SQUASHING NUMBERS TO LOG-SPACE:

Inspired by psychological evidence that our brain compresses large quantities nonlinearly using a logarithmic scale on the mental number line (Nieder & Miller, 2003; Dehaene, 2011) , we design the following squashing function to transform all the numbers in X into the log-space before prototype induction. Alternatively, we can apply the function only in the similarity function. Besides the psychological motivation, squashing is also necessary for our methods to avoid overflow during training when there are very large numbers such as 10 15 in the training corpus.

f (x) = log(x) + 1, if x > 1 x, if x ∈ [-1, 1] -log(-x) -1, if x < -1 (2) 3.2 PROTOTYPE INDUCTION

We develop two methods for inducing a small prototype set P from the number dataset X. Denote the number of prototypes by m.

Self-Organizing Map A self-organizing map (SOM) (Kohonen, 1990 ) is an artificial neural network that can be viewed as a clustering method. After training a SOM on the dataset X, we regard each cluster centroid as a prototype. One advantage of using a SOM in comparison with traditional clustering methods is that it distributes prototypes more evenly on the number line and may assign prototypes to number ranges with few training samples, which we expect would lead to better generalizability.

Gaussian Mixture Model Inspired by psychological study of the mental number line (Dehaene et al., 2003) and previous work on language modeling (Spithourakis & Riedel, 2018) , we train a Gaussian mixture model (GMM) to induce number prototypes. A GMM is defined as follows.

EQUATION

where Z is a latent variable representing the mixture component for random variable U , and N is the probability density function of a normal distribution, and π k , µ k , σ k ∈ R represent the mixing coefficient, mean and standard deviation of the k-th Gaussian component. We train a GMM on the number dataset X using the expectation-maximization (EM) or hard-EM algorithm and regard the means of the learned Gaussian components as our prototypes P = {µ 1 , • • • , µ m }. We use three GMM initialization methods described in Appendix A.

3.3. SIMILARITY FUNCTION:

For SOM-induced prototypes, we define the following similarity function:

sim(p, n) = |g(p) -g(n)| -β , β > 0, p ∈ P (4)

where function g is equal to the squashing function f defined in Eq.2 if we do not apply log transformation before prototype induction and is the identity function I otherwise. β is a hyper-parameter set to 1.0 by default.

For GMM-induced prototypes, we can naturally use the posterior probability of the component assignment to define the similarity function.

sim(p k , n) ∝ P (Z = k|U = n) = π k N (n; µ k , σ 2 k ) m k=1 π k N (n; µ k , σ 2 k )

, p k ∈ P (5)

Figure 1 : The computational graph when the center word is 'is' and the context words are 'he' and the numeral '190'. We look up the embedding vectors of non-numerical words directly from the embedding matrices and use the weighted average of prototype embeddings as the numeral embedding. Negative sampling is not shown in the figure.

3.4. EMBEDDING TRAINING:

We now describe how to integrate our numeral embeddings into traditional word embedding methods for training. We choose skip-gram with negative sampling (Mikolov et al., 2013a; b) as the word embedding method here, but many other word embedding methods such as CBOW (Mikolov et al., 2013a) , HAL (Lund & Burgess, 1996) and GloVe (Pennington et al., 2014) can be used as well.

Skip-gram is a word embedding method based on the idea of context word prediction. The training corpus C is regarded as a sequence of words (x 1 , . . . , x T ). For token x t , we define the preceding and following c tokens as the context of x t . Skip-gram aims to maximize p(x t+j |x t ) (-c ≤ j ≤ c), the probability of a context word given the center word x t . To formulate p(x t+j |x t ), skip-gram associates each word x i with two vector representations: the input embedding v i xt for being a center word and the output embedding v o xt for being a context word. The input and output embeddings of all the words in the vocabulary V constitute matrices E I ∈ R D×|V| and E O ∈ R D×|V| respectively, where D is the dimension of word embeddings. The conditional probability p(x t+j |x t ) is then defined to based on the dot product s(x

t+j |x t ) = v i xt T v o xt+j .

Nagative sampling is used to approximate the normalization factor for the conditional probability.

EQUATION

where σ denotes the sigmoid function, and P n (x) is the negative word sampling distribution used to draw k negative samples.

We modify skip-gram by computing numeral embeddings differently from non-numerical word embeddings. We associate each prototype number with an input embedding and an output embedding. The input and output embeddings of all the prototypes constitute matrices M I ∈ R D×|P| and M O ∈ R D×|P| respectively. For any numeral, we can compute its input and output embeddings by taking a weighted average of the prototype input and output embeddings respectively based on Eq.1 and use them in exactly the same way as the embeddings of non-numerical words to compute the learning objective (Eq.6). When drawing negative samples, we first set the ratio of numerals and non-numerical words to their actual ratio in the training corpus, to guarantee a sufficient number of numeral negative samples. Then we sample numerals and non-numerical words separately from their respective distributions in the training corpus raised to the power of 3 4 . During training, we optimize the objective function Eq.6 by back-propagating the gradient of error to update both the embedding matrices both the non-numerical word embedding matrices E I , E O and the prototype number embedding matrices M I , M O . In this way, the embeddings of non-numerical words and numerals are learned jointly in the same space. We show an example in Figure 1 . 

4. EXPERIMENTS AND RESULTS:

We evaluate our methods on four intrinsic and extrinsic tasks: word similarity, embedding numeracy, numeral prediction, and sequence labeling. We report results of our methods based on SOM and GMM separately. We choose the hyper-parameters (e.g., the number of prototypes, GMM initialization and training methods) using validation sets and report the best hyper-parameters for each experiment in Appendix B.

4.1. BASELINES:

NumAsTok This baseline treats numerals and non-numerical words in the same way, which is very similar to the original skip-gram. The vocabulary includes both high-frequency words and highfrequency numerals. OOV non-numerical words are replaced with symbol UNK word and OOV numerals are replaced with symbol UNK num . D-LSTM Character-level RNNs are often used to encode OOV words (Graves, 2013) . Here we apply an LSTM (Hochreiter & Schmidhuber, 1997) to the digit sequence of a numeral and use the last hidden state of the LSTM as the embedding of the numeral. We use the embedding to compute the skip-gram objective function and propagate the gradients back to update the LSTM.

The vocabulary of digits is: {0-9, '.', '+', '-', 'e'}. Fixed This baseline fixed embeddings for numerals with no training. We define the embedding a numeral with value n as [f (n); 1]/Z where f is the squashing function defined in Eq.2, 1 ∈ R D-1 is an all-ones vector, and Z is a constant used to keep the vector norm close to those of non-numerical words and is set to 2 × D by default.

We compare the vocabularies of different methods in Table 1 . Our methods, D-LSTM, and Fixed have finite non-numerical vocabularies but infinite numeral vocabularies. In contrast, the NumAsTok baseline has a finite numeral vocabulary and treats all the OOV numerals as UNK num .

4.2. WORD SIMILARITY FOR NON-NUMERICAL WORDS:

To ensure that our methods can still generate high quality embeddings for non-numerical words, we evaluate our trained embeddings on classical intrinsic word similarity tasks, including WordSim-353, (Finkelstein et al., 2001) , MEN (Bruni et al., 2014) and Simplex-999 (Hill et al., 2014) . We train 300-dimensional word embeddings on the 1B Wikipedia dump and set the context window size to 5, the number of negative samples to 5, and the vocabulary size to 3 × 10 5 . We use the evaluation tools 1 provided by Jastrzebski (Jastrzebski et al., 2017) . Note that while the training data contains numerals, the evaluation tasks do not involve numerals and are only designed to evaluate quality of non-numerical word embeddings. The results are shown in Table 2 .

It can be seen that our methods can achieve scores comparable to those of the baselines. The performance of SG trained on 100B GoogleNews is much better than all the other methods probably because of its much larger training corpus. The results show that adding our numeral embedding methods into skip-gram does not harm the quality of non-numerical word embeddings. Additional results of our methods can be found in Appendix C. 2019) propose a framework for evaluating the ability of numeral embeddings to capture magnitude and numeration. Given a target numeral, its embedding is evaluated against a set of numerals using the OVA (One-vs-All), SC (Strict Contrastive) and BC (Broad Contrastive) tests:

• OVA: The embedding vector distance between the target and its nearest neighbor on the number line should be smaller than that between the target and any other numeral in the set.

• SC: The embedding vector distance between the target and its nearest neighbor on the number line should be smaller than that between the target and its second nearest neighbors on the number line.

• BC: The embedding vector distance between the target and its nearest neighbor on the number line should be smaller than that between the target and its furthest neighbors on the number line.

We follow the settings described by Naik et al. (2019) : for the magnitude evaluation, we run the tests using a set of 2342 numerals that are most frequent in Wikipedia-1B, whose embeddings are well learned by all the methods; and for the numeration evaluation, we run the tests using 113 English words that represent numbers (e.g., 'three', 'billion') sampled from the same corpus and we measure the distance between the target numeral embedding and the word embeddings of these words. We report the accuracy of various embedding models on these three tests, along with the average rank (denoted as AVGR) of the target numeral's nearest neighbor among all the candidates based on their vector distances to the target. We use the embeddings trained on Wikipedia-1B.

Table 3 shows the results. The Fixed baseline has the best performance in the magnitude evaluation, which is unsurprising because the numeral embedding vector explicitly contains the (squashed) magnitude. NumAsTok performs very well in the numeration evaluation, which is because the number-representing words used in the evaluation are high-frequency words and their embeddings are adequately trained. Except for these two special cases, our methods can be seen to outperform the baselines with a large margin. Wallace et al. (2019) recently show that classic embeddings of numerals may contain magnitude information that can be extracted by neural networks. Following their methodology, we conduct two probing tests on our 2342 numerals using multi-layer perceptrons and bilinear functions and then use the resulting models to predict distances between numerals in the OVA, SC, and BC tasks. The results again show the advantage of our methods over the baselines. See Appendix D for details.

4.4. NUMERAL PREDICTION:

To evaluate the quality of numeral embeddings, we design a new numeral prediction task: choosing the right numeral from a set of candidates given the context of the numeral in a sentence.

We randomly sample 2000 sentences containing numerals from a subset of Wikipedia that is not used in training, with 600 for validation and 1400 for testing. For each sentence, we use the five words preceding and following the target numeral as its context. An example is shown below, where the ten bold words are the context and 2.31 is the target numeral.

In Hollywood, the average household size was [2.31] and the average family size was 3.00. 

) = v o n T v i cj .

Our first candidateranking score function S A is the sum of log probabilities of center numeral n predicting each context word c j . We use softmax here to calculate the probability.

S A (n) = j log p(cj|n) ≈ j log e s(c j |n) c k ∈V t e s(c k |n) = j s(cj|n) - j log Z(n) (7)

where V t is the vocabulary of non-numerical words and Z(n) is the normalization factor. The other candidate-ranking score function S B is the sum of log probabilities of each context word c j predicting center numeral n.

S B (n) = j log p(n|cj) ≈ j log e s(n|c j ) n k ∈Vn e s(n k |c j ) = j s(n|cj) -Constant (8)

where V n is the set of numerals in the dataset. There are a few other possible score functions, but we find that they lead to results similar to S A and S B .

We use three metrics to evaluate numeral prediction (Spithourakis & Riedel, 2018) . MdAE is the median of the absolute errors between the predicted and true numerals, MdAPE is the median of the absolute percentage errors between the predicted and true numerals, and AVGR is the average rank of the true numeral among the candidates. Detailed formulas of the three metrics are shown in Appendix E.

We train embeddings on Wikipedia-1B and report the evaluation results in the left part of Table 4 . Our methods significantly outperform the NumAsTok and Fixed baselines on all the three metrics. D-LSTM also performs well but needs more parameters and computing time than our methods.

We also conduct a slightly different numeral prediction task on the recently released Numeracy-600K dataset (the Article Title part) (Chen et al., 2019) . This dataset contains 600k sentences with numerals and in each sentence, one numeral is selected and tagged with its order of magnitude.

There are eight possible orders of magnitude and the goal is to predict the correct one for the target numeral from its context. To solve this multi-class classification problem, we sample 100 numerals for each order of magnitude and use the mean of their numeral embeddings to create a 'meta' embedding; we then use these 'meta' embeddings to replace the numeral embeddings in the score functions S A and S B and the highest-scoring order of magnitude is returned.

We split the dataset to 450k sentences for training, 50k for validation and 100k for testing. We use micro-F1 and macro-F1 in addition to AVGR as the evaluation metrics. The result is shown in the right part of Table 4 . The result shows that our methods achieve much better performance compared to the baselines.

4.5. SEQUENCE LABELING ON CUSTOMER SERVICE DATA:

To verify the effectiveness of our methods in practice, we evaluate our methods with a sequence labeling task on a dataset of customer service chat log from an online apparel shopping website. Table 5 : The results of sequence labeling. We report the accuracy, precision, recall, F1 score for the original, augmented, and harder test sets with different training data sizes. Accuracy is in the token level and the other metrics are in the entity level. We report the standard deviations in Appendix H. This dataset contains a large number of numerals related to height, weight, foot length, etc., and therefore is a good testbed for evaluating numeral embeddings.

The task is to assign a label to each word or numeral in the dataset indicating its information type. We shows two examples below: W, H, O are labels representing weight, height and ordinary word respectively. We show the statistics of the dataset in Appendix G. In order to better evaluate the generalizability, we create two additional test sets. The first one is created by 'augmenting' the original test set with new sentences containing slightly perturbed numerals. For example, we can create new sentences by replacing '177' in the above example with '176' and '178'. The second one contains 'hard' sentences from the original test set that do not have explicit cues for label prediction. For example, the first sentence above contains 'kg' and 'cm' that can greatly facilitate the prediction of W and H, but the second sentence above does not contain such cues and hence is a 'hard' sentence. More details about the two test sets can be found in Appendix F. Finally, we also test the low-resource settings in which only 30% or 10% of the training set is used.

We learn embeddings from the training set using our methods and the baselines and use a validation set to do model selection. We plug the learned embeddings into the Neural-CRF model (Yang & Zhang, 2018) 2 to do sequence labeling without using part-of-speech and character-level features and embedding fine-tuning.

The results are shown in Table 5 . Our methods consistently outperform all the baselines on the Accuracy, Recall, and F1 metrics in different configurations. NumAsTok trained with 100% training samples has the highest precision on the original and hard test sets probably because it learns highquality embeddings for high-frequency numerals included in its vocabulary; but its recall is lower than that of our methods, most likely because of its numeral OOV problem. Comparing the results on the original and augmented test sets, we see that NumAsTok shows a more significant drop in performance than the other methods, which suggests that NumAsTok does not generalize well because of the numeral OOV problem. In the low-resource settings, the advantage of our methods over the baselines becomes even larger, indicating better generalizability and less annotation required for our methods to achieve a promising performance.

5. CONCLUSION:

In this paper, we propose two novel numeral embedding methods that represent the embedding of a numeral as a weighted average of a set of prototype numeral embeddings. The methods can be integrated into traditional word embedding approaches such as skip-gram for training. We evaluate our methods on four intrinsic and extrinsic tasks, including word similarity, embedding numeracy, numeral prediction, and sequence labeling, and show that our methods can improve the performance of numeral-related tasks and has better generalizability. Our code and sample data can be found at path/to/code/.

An important future direction is to handle numeral polysemy. For example, the numeral "2019" may denote either a year or an ordinary number. One potential method is to assign a different embedding to each sense of a numeral. In this way, "2019" would have one embedding for representing a year and another for representing an ordinary quantity. The similarity function would treat different senses of a numeral differently. For example, the year sense of "2019" would be similar to the year sense of "19" but dissimilar to the sole sense of "2019.5", while the quantity sense of "2019" would be similar to that of "2019.5".

Table 6 also shows that the optimal number of prototypes is around 200-500 for the Wiki1B corpus and 10-25 for the much smaller sequence labeling dataset. As a rule of thumb, we suggest setting the number of prototypes to (log N ) 2 , where N is the number of distinct numerals in the training corpus. We show the histograms of numerals in the Wikipedia-1B dataset and the prototypes learned by SOM and GMM in Fig. 2 . It can be seen that the prototypes induced by our methods have a similar distribution compared to the original numerals.

We also show some examples of prototypes and their nearest non-numerical words in Table 8 . We use the embedding trained by the SOM model with 200 prototypes on Wikipedia-1B, and use log transformation in the similarity function.

In addition, we select several typical numerals and non-numerical words and project their embeddings to 2D using t-SNE (Maaten & Hinton, 2008) (Figure 3 ). We use embeddings learned on Wikipedia-1B corpus using the SOM and GMM methods. The examples and the figures show that our model does capture some semantic relations between numeral quantities and normal words.

We show the training speed of each embedding method on the Wikipedia-1B dataset in Table 9 . The batch size is set to 2048 for all the methods. Our methods are slower than NumAsTok but are faster than D-LSTM. 

D PROBING TESTS:

We apply two probing tests using neural network on our methods and baselines in order to compare their ability to encode magnitude information non-linear way. The first test is Decoding (predicting the numeral value from its embedding using MLP). The second is Subtraction (predicting the difference between two numerals from their embeddings using MLP or BiLinear functions). We illustrate the tasks and the models we use in Figure 4 .

We first create the datasets for the two probing tests based on the dataset from the magnitude evaluation of Section 4.3 (containing 2342 numerals). For Decoding, the dataset can be directly used.

For Subtraction, we randomly sample 10 5 pairs of numerals (n 1 , n 2 ) from the dataset and assign n 1 -n 2 as the prediction target. Following Wallace et al. (2019) , we randomly split 80% of each dataset for training and 20% for testing. We use SGD to optimize the mean square error (MSE) loss.

We report the root-mean-square error (RMSE) results for the two tasks in The results show that our two methods are significantly better than the baselines on Decoding. On Subtraction, they are better than the baselines when using BiLinear and are comparable to NumAs-Tok but much better than the other baselines when using MLP. We found that the performance is very sensitive to the neural network architecture and MLP with two hidden layers performs best.

We then use the MLP2 models trained on Subtraction to determine the distance between two numerals when conducting the magnitude evaluation of Section 4.3. The results are shown in Table 11 . The results show that our methods have better performance than the baselines overall. One interesting observation is that, although our SOM based method has worse RMSE than NumAsTok as shown in Table 10 , it outperforms NumAsTok in the magnitude evaluation.

E NUMERAL PREDICTION EVALUATION METRICS:

We denote the target numeral by n i , the numeral with the highest ranking score by ni , and the rank of the target numeral by r i . The error e i and percentage error pe i can be calculated as: 

EQUATION

F AUGMENTED AND HARD TEST SETS IN SEQUENCE LABELING:

The augmented test set is created by reasonably perturbing the numerals in a sentence. For example, for a numeral '173' that describes height, we generate new samples by changing '173' to '174' or '175' while keeping the other non-numerical words in the sentence unchanged. For a decimal such as '1.7 meters', we change it to '1.6' or '1.8'. The perturbation will not change the decimal places of numerals and will only change the quantity slightly, which makes the generated sentences reasonable.

The hard test set is created by manually collect 'hard' samples in the original test set. Hard samples do not have explicit patterns, meaning that a numeral's tag cannot be easily inferred by its adjacent words. For example, tags of numerals followed by units like 'cm', 'm', 'kg', 'years' and 'feet' can be figured out easily, so we exclude them from the hard test set. Customers are very likely to use ambiguous expressions like: 'I'm 16.5, can I buy 24?', where 16.5 is about foot length and 24 is the shoe size. These ambiguous sentences are included in the hard test set.

G STATISTICS OF SEQUENCE LABELING DATASET:

We show the statistics of the customer-service dataset in the 

:

https://github.com/kudkudak/word-embeddings-benchmarks

https://github.com/jiesutd/NCRFpp

Figures and Tables:

cm what size shall I choose 82 177 what size ?

Figure 2: Histograms of numerals and learned prototypes that range from 0 to 10 13 . The horizontal axis represents the numeral quantity and the vertical axis represents the number of occurrences, '500' means the number of prototypes, 'soft' means soft-EM.

(a) t-SNE plot for embedding trained by the SOM-based method with 200 prototypes. (b) t-SNE plot for embedding trained by the GMM-based method with 300 prototypes, random initialization and soft-EM training.

Figure 3: 2D t-SNE results for the SOM-based and GMM-based methods.

Vocabularies of different methods.

Table content: <table><tr><td/><td/><td/><td>Methods</td><td colspan="3">WS353 MEN SIM999</td></tr><tr><td/><td>Non-numerical Word Vocabulary</td><td>Numeral Vocabulary</td><td>SOM GMM</td><td>64.40 64.90</td><td>71.79 71.89</td><td>36.09 36.29</td></tr><tr><td>SOM, GMM, D-LSTM,</td><td>{In-vocab word}, UNK word</td><td>all numerals</td><td>NumAsTok D-LSTM Fixed</td><td>65.30 63.60 64.35</td><td>71.83 71.82 72.17</td><td>35.85 34.58 36.27</td></tr><tr><td>Fixed</td><td/><td/><td>SG GoogleNews-100B</td><td>70.00</td><td>74.10</td><td>44.20</td></tr><tr><td>NumAsTok</td><td>UNK word {In-vocab word},</td><td>UNKnum numerals}, {In-vocab</td><td>GloVe Wiki-6B</td><td>52.20</td><td>73.70</td><td>37.10</td></tr></table>

Results on word similarity tasks trained on Wiki-1B. For reference, we also show the results of the official skip-gram and GloVe trained on larger corpora.

Table content: <table/>

Magnitude and numeration evaluation results for our methods and baselines. Accuracies of OVA, SC and BC are expressed as percentages. Lower AVGR indicates better performance.

Table content: <table><tr><td/><td/><td colspan="2">Magnitude</td><td/><td/><td colspan="2">Numeration</td></tr><tr><td>Metrics</td><td>OVA</td><td>SC</td><td>BC</td><td colspan="2">AVGR OVA</td><td>SC</td><td>BC</td><td>AVGR</td></tr><tr><td>SOM</td><td colspan="2">67.72 71.86</td><td>99.40</td><td>15.91</td><td colspan="3">3.54 62.83 100.00</td><td>28.98</td></tr><tr><td>GMM</td><td colspan="3">57.86 58.63 100.00</td><td>1.75</td><td colspan="3">4.42 65.49 100.00</td><td>25.97</td></tr><tr><td colspan="3">NumAsTok 12.17 51.02</td><td>95.99</td><td colspan="3">144.13 7.08 61.95</td><td>99.12</td><td>27.08</td></tr><tr><td>D-LSTM</td><td>7.26</td><td>51.79</td><td>92.83</td><td colspan="3">158.82 1.77 54.87</td><td>89.38</td><td>53.55</td></tr><tr><td>Fixed</td><td colspan="3">83.90 78.22 100.00</td><td>1.17</td><td colspan="2">0.89 49.56</td><td>99.12</td><td>56.00</td></tr><tr><td colspan="5">Numbers indicating top-2 performance are highlighted.</td><td/><td/><td/></tr><tr><td colspan="6">4.3 MAGNITUDE AND NUMERATION OF EMBEDDINGS</td><td/><td/></tr><tr><td>Naik et al. (</td><td/><td/><td/><td/><td/><td/><td/></tr></table>

The results of the numeral prediction tasks.We use all the 1400 numerals in the test set as the candidates from which one has to select the right numeral for each test sentence. Given the learned word and numeral embeddings, we define two score functions to rank candidate numerals given the context. Following the skip-gram model, we first define the score of center numeral n predicting context word c j as s(cj |n) = v o cj T v in and the score of context word c j predicting the center numeral n as s(n|c j

Table content: <table><tr><td/><td/><td/><td colspan="2">Wikipedia-1B, dim 300</td><td/><td/><td colspan="3">Numeracy-600k, dim 300</td></tr><tr><td/><td/><td>SA</td><td/><td/><td>SB</td><td/><td>SA</td><td/><td>SB</td></tr><tr><td>Metrics</td><td>AVGR</td><td>MdAE</td><td colspan="2">MdAPE AVGR</td><td>MdAE</td><td colspan="2">MdAPE AVGR Micro-</td><td>Macro-</td><td>AVGR Micro-</td><td>Macro-</td></tr><tr><td/><td/><td/><td/><td/><td/><td/><td>F1</td><td>F1</td><td>F1</td><td>F1</td></tr><tr><td>SOM</td><td>381.41</td><td>825.79</td><td>0.9836</td><td>455.01</td><td>1184.60</td><td>0.9880</td><td colspan="2">2.91 37.99 13.50</td><td>2.02 42.74 13.66</td></tr><tr><td>GMM</td><td>343.50</td><td>1184.85</td><td>0.9450</td><td>444.15</td><td>1081.50</td><td>0.9866</td><td colspan="2">2.19 41.86 18.47</td><td>2.02 44.07 13.77</td></tr><tr><td colspan="2">NumAsTok 600.17</td><td>1918.00</td><td>0.9965</td><td colspan="2">600.28 32772.50</td><td>19.07</td><td>4.21 9.74</td><td>5.47</td><td>6.16 24.28 4.88</td></tr><tr><td>D-LSTM</td><td>357.45</td><td>1310.65</td><td>0.9369</td><td>466.81</td><td>1080.5</td><td>0.9908</td><td colspan="2">3.98 27.98 8.80</td><td>4.49 16.49 8.42</td></tr><tr><td>Fixed</td><td colspan="2">685.58 50371.50</td><td>42.82</td><td colspan="2">672.47 50525.00</td><td>61.59</td><td>3.23 0.00</td><td>0.01</td><td>3.23 0.00</td><td>0.00</td></tr></table>

.19 90.46 90.83 97.02 91.28 90.18 90.72 96.19 86.66 85.91 86.28 SOM 97.04 90.74 90.45 90.60 97.03 91.19 90.43 90.81 96.06 86.18 85.93 86.06 D-LSTM 96.72 89.84 88.80 89.32 96.72 90.40 88.99 89.69 95.52 84.19 83.30 83.74 Fixed 95.75 86.19 87.42 86.80 95.86 87.13 87.65 87.39 93.97 78.39 80.18 79.27 NumAsTok 96.88 91.37 89.29 90.32 96.36 90.99 87.39 89.15 96.00 87.11 85.12 86.10

Table content: <table><tr><td/><td/><td/><td/><td>Original</td><td/><td/><td colspan="2">Augmented</td><td/><td/><td/><td>Hard</td></tr><tr><td/><td/><td>Acc</td><td>P</td><td>R</td><td>F1</td><td>Acc</td><td>P</td><td>R</td><td>F1</td><td>Acc</td><td>P</td><td>R</td><td>F1</td></tr><tr><td colspan="13">100% 97.12 9130% GMM GMM 96.21 89.55 86.07 87.78 95.92 89.07 85.33 87.16 95.27 84.42 81.62 82.99 SOM 96.20 89.50 86.18 87.81 95.88 89.12 85.29 87.16 95.23 84.44 81.50 82.94 D-LSTM 95.55 86.83 83.88 85.33 95.30 86.22 83.13 84.64 94.32 80.10 78.17 79.12</td></tr><tr><td/><td>Fixed</td><td colspan="11">94.67 83.51 82.69 83.10 94.48 83.40 82.02 82.71 92.92 75.03 75.18 75.10</td></tr><tr><td/><td colspan="12">NumAsTok 95.58 89.18 83.55 86.27 94.57 88.39 79.94 83.95 94.65 84.42 79.06 81.65</td></tr><tr><td/><td>GMM</td><td colspan="11">93.43 82.36 75.01 78.51 92.78 81.48 72.85 76.92 93.19 80.26 72.71 76.30</td></tr><tr><td/><td>SOM</td><td colspan="11">93.48 82.13 75.11 78.46 92.87 80.96 73.22 76.89 93.24 79.47 73.04 76.11</td></tr><tr><td>10%</td><td>D-LSTM</td><td colspan="11">92.53 77.71 71.45 74.45 91.99 76.24 69.96 72.96 92.10 73.26 68.72 70.92</td></tr><tr><td/><td>Fixed</td><td colspan="11">91.90 75.39 71.41 73.34 91.48 73.96 70.20 72.02 91.06 69.50 67.47 68.46</td></tr><tr><td/><td colspan="12">NumAsTok 92.31 81.98 70.51 75.81 90.77 80.10 64.95 71.73 92.00 79.64 67.95 73.32</td></tr></table>

Hyper-parameter values for GMM and SOM based methods for each experiment.

Table content: <table><tr><td/><td/><td>SOM</td><td/><td>GMM</td><td/><td/></tr><tr><td/><td colspan="6">prototype number log transform stage prototype number log transform initialization EM</td></tr><tr><td>Word similarity (Wiki1B)</td><td>200</td><td>dataset</td><td>200</td><td>True</td><td>random</td><td>hard</td></tr><tr><td>Magnitude (Wiki1B)</td><td>200</td><td>dataset</td><td>300</td><td>True</td><td>random</td><td>soft</td></tr><tr><td>Numeration (Wiki1B)</td><td>300</td><td>similarity function</td><td>500</td><td>True</td><td>random</td><td>soft</td></tr><tr><td>Numeral Prediction (Wiki1B)</td><td>300</td><td>similarity function</td><td>300</td><td>False</td><td>random</td><td>hard</td></tr><tr><td>Numeral Prediction (Numeracy-600k)</td><td>50</td><td>dataset</td><td>200</td><td>False</td><td>random</td><td>hard</td></tr><tr><td>Sequence Labeling 100 %</td><td>15</td><td>dataset</td><td>30</td><td>False</td><td>random</td><td>soft</td></tr><tr><td>Sequence Labeling 30 %</td><td>10</td><td>dataset</td><td>15</td><td>False</td><td>k-means</td><td>soft</td></tr><tr><td>Sequence Labeling 10 %</td><td>25</td><td>similarity function</td><td>20</td><td>False</td><td>from-som</td><td>soft</td></tr></table>

Examples of prototypes and their nearest non-numerical words.

Table content: <table><tr><td>Method</td><td>SOM</td><td>GMM</td><td colspan="2">NumAsTok D-LSTM Fixed</td></tr><tr><td colspan="4">Speed (sent/s) 13590.93 12691.18 22907.97</td><td>8421.66</td><td>13055.08</td></tr></table>

Training speed for each methods.

Table content: <table/>



Table content: <table><tr><td/><td colspan="2">Decoding</td><td/><td>Subtraction</td><td/><td/></tr><tr><td/><td>MLP1</td><td>MLP2</td><td colspan="2">BiLinear BiLinear+MLP1</td><td>MLP1</td><td>MLP2</td></tr><tr><td>GMM</td><td>77.73</td><td>68.29</td><td>1006.66</td><td>62.04</td><td>18.10</td><td>4.40</td></tr><tr><td>SOM</td><td>289.40</td><td>182.91</td><td>2109.16</td><td>100.03</td><td>33.62</td><td>14.04</td></tr><tr><td colspan="4">NumAsTok 1027.33 1035.25 2835.32</td><td>282.32</td><td>15.52</td><td>4.66</td></tr><tr><td>D-LSTM</td><td colspan="3">1433.45 1397.00 3208.96</td><td>634.20</td><td>717.04</td><td>644.41</td></tr><tr><td>Fixed</td><td colspan="3">2220.75 2225.31 3791.96</td><td>3790.93</td><td colspan="2">3791.91 3791.98</td></tr></table>

Probing test results. MLP1 and MLP2 denote MLP with one and two hidden layers respectively.

Table content: <table/>

Results of the magnitude evaluation using MLP2 trained on the Subtraction probing task. Accuracies of OVA, SC and BC are expressed as percentages. Lower AVGR indicates better performance.

Table content: <table><tr><td/><td>OVA</td><td>SC</td><td>BC</td><td>AVGR</td></tr><tr><td>GMM</td><td colspan="3">8.03 48.76 100.00</td><td>16.45</td></tr><tr><td>SOM</td><td colspan="3">4.40 57.43 100.00</td><td>59.41</td></tr><tr><td colspan="5">NumAsTok 1.96 50.90 100.00 106.98</td></tr><tr><td>D-LSTM</td><td colspan="4">0.51 52.09 100.00 545.87</td></tr><tr><td>Fixed</td><td colspan="2">0.00 00.00</td><td>0.00</td><td>1170.68</td></tr></table>

The vocabulary is small because the dataset is confined to a specific domain: online customer service chat log about apparel purchase. In this dataset, most of the sentences are about sizes of various kinds of clothes and are very short and ambiguous.H SEQUENCE LABELING RESULT WITH STANDARD DEVIATION.

Table content: <table><tr><td/><td/><td>Number of Sentences</td><td/><td/></tr><tr><td>Train</td><td>Dev</td><td>Original Test</td><td colspan="2">Augmented Test Hard Test</td></tr><tr><td>1389</td><td>793</td><td>1802</td><td>8052</td><td>726</td></tr><tr><td/><td/><td colspan="2">Statistics of Training Set</td><td/></tr><tr><td colspan="3">Token Vocab Numeral Vocab Avg sent length</td><td>Numeral Ratio</td><td>labels</td></tr><tr><td>505</td><td>234</td><td>10.42</td><td>15.89 %</td><td>21</td></tr></table>

Statistics of low-resource customer-service dataset.

Table content: <table><tr><td/><td/><td/><td colspan="2">Original</td><td/><td/><td colspan="2">Augmented</td><td/><td/><td colspan="2">Hard</td><td/></tr><tr><td/><td/><td>Acc</td><td>P</td><td>R</td><td>F1</td><td>Acc</td><td>P</td><td>R</td><td>F1</td><td>Acc</td><td>P</td><td>R</td><td>F1</td></tr><tr><td/><td>GMM</td><td>97.12 ±0.05</td><td>91.19 ±0.11</td><td>90.46 ±0.17</td><td>90.83 ±0.14</td><td>97.02 ±0.06</td><td>91.28 ±0.17</td><td>90.18 ±0.19</td><td>90.72 ±0.18</td><td>96.19 ±0.05</td><td>86.66 ±0.13</td><td>85.91 ±0.36</td><td>86.28 ±0.24</td></tr><tr><td>100%</td><td>SOM</td><td>97.04 ±0.04</td><td>90.74 ±0.15</td><td>90.45 ±0.10</td><td>90.60 ±0.12</td><td>97.03 ±0.03</td><td>91.19 ±0.14</td><td>90.43 ±0.11</td><td>90.81 ±0.13</td><td>96.06 ±0.09</td><td>86.18 ±0.14</td><td>85.93 ±0.32</td><td>86.06 ±0.23</td></tr><tr><td/><td>D-LSTM</td><td>96.72 ±0.06</td><td>89.84 ±0.26</td><td>88.80 ±0.24</td><td>89.32 ±0.25</td><td>96.72 ±0.07</td><td>90.40 ±0.29</td><td>88.99 ±0.23</td><td>89.69 ±0.26</td><td>95.52 ±0.14</td><td>84.19 ±0.66</td><td>83.30 ±0.55</td><td>83.74 ±0.60</td></tr><tr><td/><td>Fixed</td><td>95.75 ±0.11</td><td>86.19 ±0.38</td><td>87.42 ±0.21</td><td>86.80 ±0.29</td><td>95.86 ±0.10</td><td>87.13 ±0.25</td><td>87.65 ±0.28</td><td>87.39 ±0.26</td><td>93.97 ±0.16</td><td>78.39 ±0.46</td><td>80.18 ±0.43</td><td>79.27 ±0.45</td></tr><tr><td/><td>NumAsTok</td><td>96.88 ±0.07</td><td>91.37 ±0.40</td><td>89.29 ±0.09</td><td>90.32 ±0.21</td><td>96.36 ±0.05</td><td>90.99 ±0.41</td><td>87.39 ±0.09</td><td>89.15 ±0.20</td><td>96.00 ±0.10</td><td>87.11 ±0.52</td><td>85.12 ±0.03</td><td>86.10 ±0.26</td></tr><tr><td/><td>GMM</td><td>96.21 ±0.07</td><td>89.55 ±0.15</td><td>86.07 ±0.32</td><td>87.78 ±0.24</td><td>95.92 ±0.09</td><td>89.07 ±0.32</td><td>85.33 ±0.40</td><td>87.16 ±0.36</td><td>95.27 ±0.13</td><td>84.42 ±0.41</td><td>81.62 ±0.47</td><td>82.99 ±0.43</td></tr><tr><td>30%</td><td>SOM</td><td>96.20 ±0.03</td><td>89.50 ±0.17</td><td>86.18 ±0.29</td><td>87.81 ±0.08</td><td>95.88 ±0.08</td><td>89.12 ±0.16</td><td>85.29 ±0.41</td><td>87.16 ±0.13</td><td>95.23 ±0.02</td><td>84.44 ±0.30</td><td>81.50 ±0.22</td><td>82.94 ±0.09</td></tr><tr><td/><td>D-LSTM</td><td>95.55 ±0.08</td><td>86.83 ±0.29</td><td>83.88 ±0.36</td><td>85.33 ±0.32</td><td>95.30 ±0.12</td><td>86.22 ±0.41</td><td>83.13 ±0.50</td><td>84.64 ±0.46</td><td>94.32 ±0.07</td><td>80.10 ±0.33</td><td>78.17 ±0.39</td><td>79.12 ±0.35</td></tr><tr><td/><td>Fixed</td><td>94.67 ±0.06</td><td>83.51 ±0.21</td><td>82.69 ±0.18</td><td>83.10 ±0.12</td><td>94.48 ±0.08</td><td>83.40 ±0.23</td><td>82.02 ±0.26</td><td>82.71 ±0.17</td><td>92.92 ±0.05</td><td>75.03 ±0.06</td><td>75.18 ±0.38</td><td>75.10 ±0.17</td></tr><tr><td/><td>NumAsTok</td><td>95.58 ±0.03</td><td>89.18 ±0.25</td><td>83.55 ±0.31</td><td>86.27 ±0.10</td><td>94.57 ±0.07</td><td>88.39 ±0.40</td><td>79.94 ±0.16</td><td>83.95 ±0.21</td><td>94.65 ±0.03</td><td>84.42 ±0.39</td><td>79.06 ±0.23</td><td>81.65 ±0.10</td></tr><tr><td/><td>GMM</td><td>93.43 ±0.12</td><td>82.36 ±0.17</td><td>75.01 ±0.52</td><td>78.51 ±0.21</td><td>92.78 ±0.03</td><td>81.48 ±0.25</td><td>72.85 ±0.36</td><td>76.92 ±0.14</td><td>93.19 ±0.04</td><td>80.26 ±0.41</td><td>72.71 ±0.09</td><td>76.30 ±0.19</td></tr><tr><td>10%</td><td>SOM</td><td>93.48 ±0.11</td><td>82.13 ±0.26</td><td>75.11 ±0.41</td><td>78.46 ±0.21</td><td>92.87 ±0.10</td><td>80.96 ±0.25</td><td>73.22 ±0.37</td><td>76.89 ±0.19</td><td>93.24 ±0.10</td><td>79.47 ±0.07</td><td>73.04 ±0.49</td><td>76.11 ±0.30</td></tr><tr><td/><td>D-LSTM</td><td>92.53 ±0.19</td><td>77.71 ±0.38</td><td>71.45 ±0.82</td><td>74.45 ±0.61</td><td>91.99 ±0.26</td><td>76.24 ±0.40</td><td>69.96 ±0.11</td><td>72.96 ±0.80</td><td>92.10 ±0.16</td><td>73.26 ±0.26</td><td>68.72 ±0.70</td><td>70.92 ±0.40</td></tr><tr><td/><td>Fixed</td><td>91.90 ±0.05</td><td>75.39 ±0.46</td><td>71.41 ±0.58</td><td>73.34 ±0.17</td><td>91.48 ±0.12</td><td>73.96 ±0.64</td><td>70.20 ±0.73</td><td>72.02 ±0.40</td><td>91.06 ±0.06</td><td>69.50 ±0.78</td><td>67.47 ±0.27</td><td>68.46 ±0.25</td></tr><tr><td/><td>NumAsTok</td><td>92.31 ±0.12</td><td>81.98 ±0.44</td><td>70.51 ±0.56</td><td>75.81 ±0.29</td><td>90.77 ±0.14</td><td>80.10 ±0.65</td><td>64.95 ±0.66</td><td>71.73 ±0.23</td><td>92.00 ±0.06</td><td>79.64 ±0.64</td><td>67.95 ±0.38</td><td>73.32 ±0.20</td></tr></table>

The results of sequence labeling. We report the accuracy, precision, recall, F1 score for the original, augmented, and harder test sets with different training data sizes. Accuracy is in the token level and the other metrics are in the entity level.

Table content: <table/>

Peer review (iclr2020):
The paper proposes a novel method for embedding numerals which can be learned by using neural word embedding learning techniques. The paper motivates the work by reviewing the difficulty of embedding components to represent numerals: OOV in most cases. Their main contribution is the introduction of a method composes numeral embedding by a weighted average of prototype embeddings based on the similarities between the numeral and prototypes. There are two proposed prototypes: SOM and GMM and the similarity functions are an absolute difference and the density function respectively. During the training, the numerals have the proposed embeddings while the others have normal word embeddings. The paper slightly modifies the negative sampling to ensure numerals being sampled. A series of 4 empirical studies have been presented. First, the paper confirms that the proposed method does not negatively affect non-numeral embeddings. And then, the quality of the numeral embeddings are evaluated and compared. The experiments show that the proposed method has better performance on numerical property tests, numeral prediction, and a sequence labeling task. Overall, this paper has a novel contribution. The proposed method is well motivated and quite justified by the experiments abliet lacking comparison with previous published results. However, it has some weaknesses. For the method part, one of the limitations is that it is not an end-to-end method and requires a regular expression to identify the numeral. Second, the weighted average of the prototypes is reasonable, but the similarity function only relies on the magnitude. I think there are other aspects of numerical tokens that it might not be able to capture (e.g. “2019” is similar to “19” in some context). In terms of training, I think it is not hard to extend the method to full language model training (using softmax). However, adding all numerals to the vocabulary would add significant overhead. For the experiments, some design decisions are left unjustified. For example, a simple ablation experiment on the squashing function will be helpful. Furthermore, guidelines or empirical results on the effect of the number of prototypes can increase the impact of the paper. Finally, I think an analysis of the performance of numerical types will be helpful for future works (e.g., dates, phone numbers, currencies, etc, or discrete vs continuous). Minor comments and questions: 1. The log sigmoid in equation 6 is a bit strange, isn’t it log sum exp(). https://arxiv.org/pdf/1402.3722v1.pdf 2. Why do you create a new dataset for the experiment in section 4.3? 3. In section 4.4, you rank only numerals in the test set, but the scores are computed based on all numerals in the vocab. Do you have the performance of ranking all numerals? 4. Just to confirm the “training” in section 4.4 refers to learning the embedding using the skip-gram model, right?