Year: 2019

Abstract:
Word embedding is an essential building block for deep learning methods for natural language processing. Although word embedding has been extensively studied over the years, the problem of how to effectively embed numerals, a special subset of words, is still underexplored. Existing word embedding methods do not learn numeral embeddings well because there are an infinite number of numerals and their individual appearances in training corpora are highly scarce. In this paper, we propose two novel numeral embedding methods that can handle the out-ofvocabulary (OOV) problem for numerals. We first induce a finite set of prototype numerals using either a self-organizing map or a Gaussian mixture model. We then represent the embedding of a numeral as a weighted average of the prototype number embeddings. Numeral embeddings represented in this manner can be plugged into existing word embedding learning approaches such as skip-gram for training. We evaluated our methods and showed its effectiveness on four intrinsic and extrinsic tasks: word similarity, embedding numeracy, numeral prediction, and sequence labeling.

heading: 1 INTRODUCTION
text: Word embeddings, the distributed vector representations of words, have become the essential building block for deep learning approaches to natural language processing (NLP). The quality of pretrained word embeddings has been shown to significantly impact the performance of neural approaches to a variety of NLP tasks. Over the past two decades, significant progress has been made in the development of word embedding techniques (Lund & Burgess, 1996; Bengio et al., 2003; Bullinaria & Levy, 2007; Mikolov et al., 2013b; Pennington et al., 2014). However, existing word embedding methods do not handle numerals adequately and cannot directly encode the numeracy and magnitude of a numeral (Naik et al., 2019). Most methods have a limited vocabulary size and therefore can only represent a small subset of the infinite number of numerals. Furthermore, most numerals have very scarce appearances in training corpora and therefore are more likely to be outof-vocabulary (OOV) compared to non-numerical words. For example, numerals account for 6.15% of all unique tokens in English Wikipedia, but in GloVe Pennington et al. (2014) which is partially trained on Wikipedia, only 3.79% of its vocabulary is numerals. Previous work (Spithourakis et al., 2016) also shows that the numeral OOV problem is even more severe when learning word embeddings from corpora with abundant numerals such as clinical reports. Even if a numeral is included in the vocabulary, its scarcity in the training corpus would negatively impact the learning accuracy of its embedding.
The inadequate handling of numerals in existing word embedding methods can be problematic in scenarios where numerals convey critical information. Take the following sentences for example,
“Jeff is 190, so he should wear size XXL” (190 is a reasonable height for size XXL) “Jeff is 160, so he should wear size XXL” (160 is an unreasonable height for size XXL) “Jeff is 10, so he should wear size XS” (10 is an age instead of a height)
If the numerals in the example are OOV or their embeddings are not accurately learned, then it becomes impossible to judge the categories of the numerals or the reasonableness of the sentences.
In this paper, we propose two novel methods that can produce reasonable embeddings for any numerals. The key idea is to represent the embedding of a numeral as a weighted average of a small set of prototype number embeddings. The prototype numerals are induced from the training corpus
using either a self-organizing map (Kohonen, 1990) or a Gaussian mixture model. The weights are computed based on the differences between the target numeral and the prototype numerals, reflecting the inductive bias that numerals with similar quantities are likely to convey similar semantic information and thus should have similar embeddings. Numeral embeddings represented in this manner can then be plugged into a traditional word embedding method for training. We empirically evaluate our methods on four tasks: word similarity, embedding numeracy, numeral prediction, and sequence labeling. The results show that our methods can produce high-quality embeddings for both numerals and non-numerical words and improve the performance of downstream tasks.

heading: 2 RELATED WORK
text: Word Embedding Word embeddings are vector representations of words which carry semantic meanings implicitly and are trained without supervision. Most existing word embedding training methods can be divided into two classes. The first class of methods (Lund & Burgess, 1996; Rohde et al., 2006; Bullinaria & Levy, 2007; Lebret & Lebret, 2013) extract word co-occurrence statistics from the training corpus, compute a word-word matrix based on measures such as PPMI, and then apply dimension reduction techniques such as principle component analysis to produce a low-dimensional vector representation for each word. The second class of methods (Bengio et al., 2003; Collobert & Weston, 2008; Mikolov et al., 2013a;b) use a simple neural network to model the relation between a word and its context within a sliding window in the training corpus. GloVe (Pennington et al., 2014) has been proposed as a method that combines the advantages of both classes. All the above methods have a finite vocabulary size and use a ‘UNK’ symbol to represent OOV words. Recent work (Naik et al., 2019) shows that these popular methods do not handle numerals adequately. Wallace et al. (2019) shows that existing word embedding methods can encode numeracy implicitly for high-frequency numerals, but the embedding’s numeracy for OOV numerals is not investigated. Our goal is to design better numeral embedding methods that can be integrated into traditional word embedding methods and handle the OOV problem for numerals.
Numeracy in natural language Numeral understanding has been found important in textual entailment (Lev et al., 2004; De Marneffe et al., 2008; Roy et al., 2015) and information extraction (Intxaurrondo et al., 2015; Madaan et al., 2016), but existing systems often use manually defined task-specific features and logic rules to identify numerals, which is hard to generalize to other tasks. A lot of research has been done trying to solve math problems, using either manually designed features and rules (Roy et al., 2015; Mitra & Baral, 2016; Roy & Roth, 2016; Upadhyay et al., 2016) or sequence-to-sequence neural networks Wang et al. (2017), but the quantity of numerals is not important in this task and hence existing methods often replace numerals by dummy symbols such as n1 and n2. Spithourakis & Riedel (2018) studied different strategies to better model numerals in language models. Chen et al. (2019) created Numeracy-600K dataset and studied the ability of neural network models to learn numeracy. Our work differs from previous work in that we aim to produce general-purpose numeral embeddings that can be employed in any neural NLP approach.

heading: 3 METHODS
text: Given a training corpus C, we first extract all the numerals using regular expressions and form a dataset X containing all the numbers represented by these numerals. A number (e.g., 2000) may appear for multiple times in X if its corresponding numerals (e.g., ‘2000’, ‘2,000’, etc.) appear for multiple times in C. We then induce a finite set P of typical numerals (i.e., prototypes) from X using a self-organizing map (Kohonen, 1990) or a Gaussian mixture model. We also define a function sim(n1, n2) outputting the similarity between two arbitrary numbers n1 and n2. Now we represent the embedding of any target numeral n as a weighted average of the prototype number embeddings with the weights computed by the similarity function:
e(n) = · ∑ p∈P α · sim(n, p) · e(p), ∑ p∈P α · sim(n, p) = 1 (1)
We use e(·) to denote the embedding of a number α is the normalization factor. This formulation satisfies the intuition that numerals with similar quantities are likely to convey similar semantic information and thus should have similar embeddings.
Our numeral embeddings can be integrated into traditional word embedding methods such as skipgram for training. During training, we back-propagate the error gradient to update the prototype number embeddings. In this way, the prototype number embeddings (and hence all the numeral embeddings) are learned jointly with non-numerical word embeddings.

heading: 3.1 SQUASHING NUMBERS TO LOG-SPACE
text: Inspired by psychological evidence that our brain compresses large quantities nonlinearly using a logarithmic scale on the mental number line (Nieder & Miller, 2003; Dehaene, 2011), we design the following squashing function to transform all the numbers in X into the log-space before prototype induction. Alternatively, we can apply the function only in the similarity function. Besides the psychological motivation, squashing is also necessary for our methods to avoid overflow during training when there are very large numbers such as 1015 in the training corpus.
f(x) = { log(x) + 1, if x > 1 x, if x ∈ [−1, 1] − log(−x)− 1, if x < −1
(2)

heading: 3.2 PROTOTYPE INDUCTION
text: We develop two methods for inducing a small prototype set P from the number dataset X . Denote the number of prototypes by m.
Self-Organizing Map A self-organizing map (SOM) (Kohonen, 1990) is an artificial neural network that can be viewed as a clustering method. After training a SOM on the dataset X , we regard each cluster centroid as a prototype. One advantage of using a SOM in comparison with traditional clustering methods is that it distributes prototypes more evenly on the number line and may assign prototypes to number ranges with few training samples, which we expect would lead to better generalizability.
Gaussian Mixture Model Inspired by psychological study of the mental number line (Dehaene et al., 2003) and previous work on language modeling (Spithourakis & Riedel, 2018), we train a Gaussian mixture model (GMM) to induce number prototypes. A GMM is defined as follows.
p(U = n) = m∑ k=1 P (Z = k)P (U = n|Z = k) = m∑ k=1 πkN (n;µk, σ2k) (3)
where Z is a latent variable representing the mixture component for random variable U , and N is the probability density function of a normal distribution, and πk, µk, σk ∈ R represent the mixing coefficient, mean and standard deviation of the k-th Gaussian component. We train a GMM on the number dataset X using the expectation-maximization (EM) or hard-EM algorithm and regard the means of the learned Gaussian components as our prototypes P = {µ1, · · · , µm}. We use three GMM initialization methods described in Appendix A.

heading: 3.3 SIMILARITY FUNCTION
text: For SOM-induced prototypes, we define the following similarity function:
sim(p, n) = |g(p)− g(n)|−β , β > 0, p ∈ P (4)
where function g is equal to the squashing function f defined in Eq.2 if we do not apply log transformation before prototype induction and is the identity function I otherwise. β is a hyper-parameter set to 1.0 by default.
For GMM-induced prototypes, we can naturally use the posterior probability of the component assignment to define the similarity function.
sim(pk, n) ∝ P (Z = k|U = n) = πkN (n;µk, σ2k)∑m k=1 πkN (n;µk, σ2k) , pk ∈ P (5)

heading: 3.4 EMBEDDING TRAINING
text: We now describe how to integrate our numeral embeddings into traditional word embedding methods for training. We choose skip-gram with negative sampling (Mikolov et al., 2013a;b) as the word embedding method here, but many other word embedding methods such as CBOW (Mikolov et al., 2013a), HAL (Lund & Burgess, 1996) and GloVe (Pennington et al., 2014) can be used as well.
Skip-gram is a word embedding method based on the idea of context word prediction. The training corpus C is regarded as a sequence of words (x1, . . . , xT ). For token xt, we define the preceding and following c tokens as the context of xt. Skip-gram aims to maximize p(xt+j |xt) (−c ≤ j ≤ c), the probability of a context word given the center word xt. To formulate p(xt+j |xt), skip-gram associates each word xi with two vector representations: the input embedding vixt for being a center word and the output embedding voxt for being a context word. The input and output embeddings of all the words in the vocabulary V constitute matrices EI ∈ RD×|V| and EO ∈ RD×|V| respectively, where D is the dimension of word embeddings. The conditional probability p(xt+j |xt) is then defined to based on the dot product s(xt+j |xt) = vixt T voxt+j . Nagative sampling is used to approximate the normalization factor for the conditional probability.
log p(xt+j |xt) ≈ log σ(voxt+j T vixt) + k∑ i=1 E xi∼Pn(x) [log σ(−voxi T vixt)] (6)
where σ denotes the sigmoid function, and Pn(x) is the negative word sampling distribution used to draw k negative samples.
We modify skip-gram by computing numeral embeddings differently from non-numerical word embeddings. We associate each prototype number with an input embedding and an output embedding. The input and output embeddings of all the prototypes constitute matrices MI ∈ RD×|P| and MO ∈ RD×|P| respectively. For any numeral, we can compute its input and output embeddings by taking a weighted average of the prototype input and output embeddings respectively based on Eq.1 and use them in exactly the same way as the embeddings of non-numerical words to compute the learning objective (Eq.6). When drawing negative samples, we first set the ratio of numerals and non-numerical words to their actual ratio in the training corpus, to guarantee a sufficient number of numeral negative samples. Then we sample numerals and non-numerical words separately from their respective distributions in the training corpus raised to the power of 34 . During training, we optimize the objective function Eq.6 by back-propagating the gradient of error to update both the embedding matrices both the non-numerical word embedding matrices EI , EO and the prototype number embedding matrices MI , MO. In this way, the embeddings of non-numerical words and numerals are learned jointly in the same space. We show an example in Figure 1.
Methods WS353 MEN SIM999 SOM 64.40 71.79 36.09 GMM 64.90 71.89 36.29
NumAsTok 65.30 71.83 35.85 D-LSTM 63.60 71.82 34.58
Fixed 64.35 72.17 36.27 SG GoogleNews-100B 70.00 74.10 44.20
GloVe Wiki-6B 52.20 73.70 37.10
Table 2: Results on word similarity tasks trained on Wiki-1B. For reference, we also show the results of the official skip-gram and GloVe trained on larger corpora.

heading: 4 EXPERIMENTS AND RESULTS
text: We evaluate our methods on four intrinsic and extrinsic tasks: word similarity, embedding numeracy, numeral prediction, and sequence labeling. We report results of our methods based on SOM and GMM separately. We choose the hyper-parameters (e.g., the number of prototypes, GMM initialization and training methods) using validation sets and report the best hyper-parameters for each experiment in Appendix B.

heading: 4.1 BASELINES
text: NumAsTok This baseline treats numerals and non-numerical words in the same way, which is very similar to the original skip-gram. The vocabulary includes both high-frequency words and highfrequency numerals. OOV non-numerical words are replaced with symbol UNKword and OOV numerals are replaced with symbol UNKnum. D-LSTM Character-level RNNs are often used to encode OOV words (Graves, 2013). Here we apply an LSTM (Hochreiter & Schmidhuber, 1997) to the digit sequence of a numeral and use the last hidden state of the LSTM as the embedding of the numeral. We use the embedding to compute the skip-gram objective function and propagate the gradients back to update the LSTM. The vocabulary of digits is: {0-9, ‘.’, ‘+’, ‘−’, ‘e’}. Fixed This baseline fixed embeddings for numerals with no training. We define the embedding a numeral with value n as [f(n);1]/Z where f is the squashing function defined in Eq.2, 1 ∈ RD−1 is an all-ones vector, and Z is a constant used to keep the vector norm close to those of non-numerical words and is set to 2×D by default. We compare the vocabularies of different methods in Table 1. Our methods, D-LSTM, and Fixed have finite non-numerical vocabularies but infinite numeral vocabularies. In contrast, the NumAsTok baseline has a finite numeral vocabulary and treats all the OOV numerals as UNKnum.

heading: 4.2 WORD SIMILARITY FOR NON-NUMERICAL WORDS
text: To ensure that our methods can still generate high quality embeddings for non-numerical words, we evaluate our trained embeddings on classical intrinsic word similarity tasks, including WordSim353, (Finkelstein et al., 2001), MEN (Bruni et al., 2014) and Simplex-999 (Hill et al., 2014). We train 300-dimensional word embeddings on the 1B Wikipedia dump and set the context window size to 5, the number of negative samples to 5, and the vocabulary size to 3× 105. We use the evaluation tools1 provided by Jastrzebski (Jastrzebski et al., 2017). Note that while the training data contains numerals, the evaluation tasks do not involve numerals and are only designed to evaluate quality of non-numerical word embeddings. The results are shown in Table 2.
It can be seen that our methods can achieve scores comparable to those of the baselines. The performance of SG trained on 100B GoogleNews is much better than all the other methods probably because of its much larger training corpus. The results show that adding our numeral embedding methods into skip-gram does not harm the quality of non-numerical word embeddings. Additional results of our methods can be found in Appendix C.
1https://github.com/kudkudak/word-embeddings-benchmarks

heading: 4.3 MAGNITUDE AND NUMERATION OF EMBEDDINGS
text: Naik et al. (2019) propose a framework for evaluating the ability of numeral embeddings to capture magnitude and numeration. Given a target numeral, its embedding is evaluated against a set of numerals using the OVA (One-vs-All), SC (Strict Contrastive) and BC (Broad Contrastive) tests:
• OVA: The embedding vector distance between the target and its nearest neighbor on the number line should be smaller than that between the target and any other numeral in the set.
• SC: The embedding vector distance between the target and its nearest neighbor on the number line should be smaller than that between the target and its second nearest neighbors on the number line.
• BC: The embedding vector distance between the target and its nearest neighbor on the number line should be smaller than that between the target and its furthest neighbors on the number line.
We follow the settings described by Naik et al. (2019): for the magnitude evaluation, we run the tests using a set of 2342 numerals that are most frequent in Wikipedia-1B, whose embeddings are well learned by all the methods; and for the numeration evaluation, we run the tests using 113 English words that represent numbers (e.g., ‘three’, ‘billion’) sampled from the same corpus and we measure the distance between the target numeral embedding and the word embeddings of these words. We report the accuracy of various embedding models on these three tests, along with the average rank (denoted as AVGR) of the target numeral’s nearest neighbor among all the candidates based on their vector distances to the target. We use the embeddings trained on Wikipedia-1B.
Table 3 shows the results. The Fixed baseline has the best performance in the magnitude evaluation, which is unsurprising because the numeral embedding vector explicitly contains the (squashed) magnitude. NumAsTok performs very well in the numeration evaluation, which is because the number-representing words used in the evaluation are high-frequency words and their embeddings are adequately trained. Except for these two special cases, our methods can be seen to outperform the baselines with a large margin.
Wallace et al. (2019) recently show that classic embeddings of numerals may contain magnitude information that can be extracted by neural networks. Following their methodology, we conduct two probing tests on our 2342 numerals using multi-layer perceptrons and bilinear functions and then use the resulting models to predict distances between numerals in the OVA, SC, and BC tasks. The results again show the advantage of our methods over the baselines. See Appendix D for details.

heading: 4.4 NUMERAL PREDICTION
text: To evaluate the quality of numeral embeddings, we design a new numeral prediction task: choosing the right numeral from a set of candidates given the context of the numeral in a sentence.
We randomly sample 2000 sentences containing numerals from a subset of Wikipedia that is not used in training, with 600 for validation and 1400 for testing. For each sentence, we use the five words preceding and following the target numeral as its context. An example is shown below, where the ten bold words are the context and 2.31 is the target numeral.
In Hollywood, the average household size was [2.31] and the average family size was 3.00.
We use all the 1400 numerals in the test set as the candidates from which one has to select the right numeral for each test sentence. Given the learned word and numeral embeddings, we define two score functions to rank candidate numerals given the context. Following the skip-gram model, we first define the score of center numeral n predicting context word cj as s(cj |n) = vocj
T vin and the score of context word cj predicting the center numeral n as s(n|cj) = vonT vicj . Our first candidateranking score function SA is the sum of log probabilities of center numeral n predicting each context word cj . We use softmax here to calculate the probability.
SA(n) = ∑ j log p(cj |n) ≈ ∑ j log es(cj |n)∑ ck∈Vt es(ck|n) = ∑ j s(cj |n)− ∑ j logZ(n) (7)
where Vt is the vocabulary of non-numerical words and Z(n) is the normalization factor. The other candidate-ranking score function SB is the sum of log probabilities of each context word cj predicting center numeral n.
SB(n) = ∑ j log p(n|cj) ≈ ∑ j log es(n|cj)∑ nk∈Vn es(nk|cj) = ∑ j s(n|cj)− Constant (8)
where Vn is the set of numerals in the dataset. There are a few other possible score functions, but we find that they lead to results similar to SA and SB.
We use three metrics to evaluate numeral prediction (Spithourakis & Riedel, 2018). MdAE is the median of the absolute errors between the predicted and true numerals, MdAPE is the median of the absolute percentage errors between the predicted and true numerals, and AVGR is the average rank of the true numeral among the candidates. Detailed formulas of the three metrics are shown in Appendix E.
We train embeddings on Wikipedia-1B and report the evaluation results in the left part of Table 4. Our methods significantly outperform the NumAsTok and Fixed baselines on all the three metrics. D-LSTM also performs well but needs more parameters and computing time than our methods.
We also conduct a slightly different numeral prediction task on the recently released Numeracy600K dataset (the Article Title part) (Chen et al., 2019). This dataset contains 600k sentences with numerals and in each sentence, one numeral is selected and tagged with its order of magnitude. There are eight possible orders of magnitude and the goal is to predict the correct one for the target numeral from its context. To solve this multi-class classification problem, we sample 100 numerals for each order of magnitude and use the mean of their numeral embeddings to create a ‘meta’ embedding; we then use these ‘meta’ embeddings to replace the numeral embeddings in the score functions SA and SB and the highest-scoring order of magnitude is returned.
We split the dataset to 450k sentences for training, 50k for validation and 100k for testing. We use micro-F1 and macro-F1 in addition to AVGR as the evaluation metrics. The result is shown in the right part of Table 4. The result shows that our methods achieve much better performance compared to the baselines.

heading: 4.5 SEQUENCE LABELING ON CUSTOMER SERVICE DATA
text: To verify the effectiveness of our methods in practice, we evaluate our methods with a sequence labeling task on a dataset of customer service chat log from an online apparel shopping website.
This dataset contains a large number of numerals related to height, weight, foot length, etc., and therefore is a good testbed for evaluating numeral embeddings.
The task is to assign a label to each word or numeral in the dataset indicating its information type. We shows two examples below:
W O H O O O O O O W H O O O 82 kg 177 cm what size shall I choose 82 177 what size ?
W, H, O are labels representing weight, height and ordinary word respectively. We show the statistics of the dataset in Appendix G. In order to better evaluate the generalizability, we create two additional test sets. The first one is created by ‘augmenting’ the original test set with new sentences containing slightly perturbed numerals. For example, we can create new sentences by replacing ‘177’ in the above example with ‘176’ and ‘178’. The second one contains ‘hard’ sentences from the original test set that do not have explicit cues for label prediction. For example, the first sentence above contains ‘kg’ and ‘cm’ that can greatly facilitate the prediction of W and H, but the second sentence above does not contain such cues and hence is a ‘hard’ sentence. More details about the two test sets can be found in Appendix F. Finally, we also test the low-resource settings in which only 30% or 10% of the training set is used.
We learn embeddings from the training set using our methods and the baselines and use a validation set to do model selection. We plug the learned embeddings into the Neural-CRF model (Yang & Zhang, 2018) 2 to do sequence labeling without using part-of-speech and character-level features and embedding fine-tuning.
The results are shown in Table 5. Our methods consistently outperform all the baselines on the Accuracy, Recall, and F1 metrics in different configurations. NumAsTok trained with 100% training samples has the highest precision on the original and hard test sets probably because it learns highquality embeddings for high-frequency numerals included in its vocabulary; but its recall is lower than that of our methods, most likely because of its numeral OOV problem. Comparing the results on the original and augmented test sets, we see that NumAsTok shows a more significant drop in performance than the other methods, which suggests that NumAsTok does not generalize well because of the numeral OOV problem. In the low-resource settings, the advantage of our methods over the baselines becomes even larger, indicating better generalizability and less annotation required for our methods to achieve a promising performance.

heading: 5 CONCLUSION
text: In this paper, we propose two novel numeral embedding methods that represent the embedding of a numeral as a weighted average of a set of prototype numeral embeddings. The methods can be
2https://github.com/jiesutd/NCRFpp
integrated into traditional word embedding approaches such as skip-gram for training. We evaluate our methods on four intrinsic and extrinsic tasks, including word similarity, embedding numeracy, numeral prediction, and sequence labeling, and show that our methods can improve the performance of numeral-related tasks and has better generalizability. Our code and sample data can be found at path/to/code/.
An important future direction is to handle numeral polysemy. For example, the numeral “2019” may denote either a year or an ordinary number. One potential method is to assign a different embedding to each sense of a numeral. In this way, “2019” would have one embedding for representing a year and another for representing an ordinary quantity. The similarity function would treat different senses of a numeral differently. For example, the year sense of “2019” would be similar to the year sense of “19” but dissimilar to the sole sense of “2019.5”, while the quantity sense of “2019” would be similar to that of “2019.5”.

heading: A GMM INITIALIZATION
text: Both EM and hard-EM are sensitive to initialization and we use the initialization methods described in (Blömer & Bujna, 2013). We first initialize the mean µk of the k-th Gaussian component using one of the following three strategies:
Random initialization: choose µk from X randomly. This is suitable when X contains a wide range of numbers, e.g., numbers collected from Wikipedia.
SOM-based initialization: initialize µk to pk ∈ P produced by the SOM method. K-means initialization: run randomly initialized k-means on X and then use k-means centroids to initialize µk.
We then assign the data samples to their closest means. The standard deviation of the data samples assigned to the k-th mean becomes σk.

heading: B HYPER-PARAMETERS
text: We list all of the important hyper-parameters we tune for each model.
General hyper-parameters: embedding dimension, context window size, SGD learning rate, batch size, vocabulary size, etc.
SOM hyper-parameters: number of prototypes, stage of applying the log-squashing function (stage 1: before prototype induction; stage 2: only in the similarity function).
GMM hyper-parameters: number of prototypes, whether we apply the log-squashing function to the numerals, EM initialization (from SOM, random initialization, or k-means initialization), type of EM (hard-EM or soft-EM).
We show the values of the SOM and GMM hyper-parameters in Table 6 and the values of the general hyper-parameters of all the methods in Table 7. We find that the general hyper-parameters influence the performance of our methods and the baselines in the same way, so in most cases, these hyperparameters are set to be identical for all the methods. For large training corpora (Wiki1B, Numeracy600k), we use 2048 as the batch size for D-LSTM, because D-LSTM consumes much more GPU memory. We set the batch size of the other methods to 4096. For the sequence labeling tasks, because the data is relatively small and confined to a very specific domain (chat log from online apparel shops), we set a small vocabulary size of 500 for all the methods except NumAsTok and set the vocabulary size of NumAsTok to 550 to ensure that different methods have similar numbers of parameters for word embedding training. Consequently, our methods have (500 + |P|) × D parameters for word embedding training and NumAsTok has 550 × D parameters, where P is the prototype set, whose size is typically smaller than 50, and D is the embedding dimension.
Table 6 also shows that the optimal number of prototypes is around 200–500 for the Wiki1B corpus and 10–25 for the much smaller sequence labeling dataset. As a rule of thumb, we suggest setting the number of prototypes to (logN)2, where N is the number of distinct numerals in the training corpus.

heading: C MORE RESULTS ON WIKIPEDIA-1B
text: We show the histograms of numerals in the Wikipedia-1B dataset and the prototypes learned by SOM and GMM in Fig.2. It can be seen that the prototypes induced by our methods have a similar distribution compared to the original numerals.
We also show some examples of prototypes and their nearest non-numerical words in Table 8. We use the embedding trained by the SOM model with 200 prototypes on Wikipedia-1B, and use log transformation in the similarity function.
In addition, we select several typical numerals and non-numerical words and project their embeddings to 2D using t-SNE (Maaten & Hinton, 2008) (Figure 3). We use embeddings learned on Wikipedia-1B corpus using the SOM and GMM methods. The examples and the figures show that our model does capture some semantic relations between numeral quantities and normal words.
We show the training speed of each embedding method on the Wikipedia-1B dataset in Table 9. The batch size is set to 2048 for all the methods. Our methods are slower than NumAsTok but are faster than D-LSTM.

heading: D PROBING TESTS
text: We apply two probing tests using neural network on our methods and baselines in order to compare their ability to encode magnitude information in a non-linear way. The first test is Decoding (predicting the numeral value from its embedding using MLP). The second is Subtraction (predicting the difference between two numerals from their embeddings using MLP or BiLinear functions). We illustrate the tasks and the models we use in Figure 4.
We first create the datasets for the two probing tests based on the dataset from the magnitude evaluation of Section 4.3 (containing 2342 numerals). For Decoding, the dataset can be directly used. For Subtraction, we randomly sample 105 pairs of numerals (n1, n2) from the dataset and assign n1 − n2 as the prediction target. Following Wallace et al. (2019), we randomly split 80% of each dataset for training and 20% for testing. We use SGD to optimize the mean square error (MSE) loss. We report the root-mean-square error (RMSE) results for the two tasks in Table 10.
The results show that our two methods are significantly better than the baselines on Decoding. On Subtraction, they are better than the baselines when using BiLinear and are comparable to NumAsTok but much better than the other baselines when using MLP. We found that the performance is very sensitive to the neural network architecture and MLP with two hidden layers performs best.
We then use the MLP2 models trained on Subtraction to determine the distance between two numerals when conducting the magnitude evaluation of Section 4.3. The results are shown in Table 11. The results show that our methods have better performance than the baselines overall. One interesting observation is that, although our SOM based method has worse RMSE than NumAsTok as shown in Table 10, it outperforms NumAsTok in the magnitude evaluation.

heading: E NUMERAL PREDICTION EVALUATION METRICS
text: We denote the target numeral by ni, the numeral with the highest ranking score by n̂i, and the rank of the target numeral by ri. The error ei and percentage error pei can be calculated as:
ei = ni − n̂, pei = ni − n̂i ni
(9)
Then we use the median of the absolute errors, the median of the absolute percentage errors, and the average rank as the evaluation metrics.
MdAE = median{|ei|}, MdAPE = median{|pei|}, AV GR = ri (10)

heading: F AUGMENTED AND HARD TEST SETS IN SEQUENCE LABELING
text: The augmented test set is created by reasonably perturbing the numerals in a sentence. For example, for a numeral ‘173’ that describes height, we generate new samples by changing ‘173’ to ‘174’ or ‘175’ while keeping the other non-numerical words in the sentence unchanged. For a decimal such as ‘1.7 meters’, we change it to ‘1.6’ or ‘1.8’. The perturbation will not change the decimal places of numerals and will only change the quantity slightly, which makes the generated sentences reasonable.
The hard test set is created by manually collect ‘hard’ samples in the original test set. Hard samples do not have explicit patterns, meaning that a numeral’s tag cannot be easily inferred by its adjacent words. For example, tags of numerals followed by units like ‘cm’, ‘m’, ‘kg’, ‘years’ and ‘feet’ can be figured out easily, so we exclude them from the hard test set. Customers are very likely to use ambiguous expressions like: ‘I’m 16.5, can I buy 24?’, where 16.5 is about foot length and 24 is the shoe size. These ambiguous sentences are included in the hard test set.

heading: G STATISTICS OF SEQUENCE LABELING DATASET
text: We show the statistics of the customer-service dataset in the Table 12. The vocabulary is small because the dataset is confined to a specific domain: online customer service chat log about apparel purchase. In this dataset, most of the sentences are about sizes of various kinds of clothes and are very short and ambiguous.
H SEQUENCE LABELING RESULT WITH STANDARD DEVIATION.

Review summaries:
The paper talks about a recently highlighted problem in word embeddings which is their incapability to represent numerals, especially the out-of-vocabulary numerals. For addressing the problem, they propose a method that induces a finite set of prototype numerals using either self-organizing map or Gaussian Mixture model. Then, each numeral is represented as a weighted average of prototype numeral embeddings. The method also involves squashing large quantities using log function. Finally, the training is performed similar to Skip-gram in word2vec but with the embedding of numerals computed using prototype numerals. 

The paper proposes a novel method for embedding numerals which can be learned by using neural word embedding learning techniques. The paper motivates the work by reviewing the difficulty of embedding components to represent numerals: OOV in most cases. Their main contribution is the introduction of a method composes numeral embedding by a weighted average of prototype embeddings based on the similarities between the numeral and prototypes. There are two proposed prototypes: SOM and GMM and the similarity functions are an absolute difference and the density function respectively. During the training, the numerals have the proposed embeddings while the others have normal word embeddings. The paper slightly modifies the negative sampling to ensure numerals being sampled. A series of 4 empirical studies have been presented. First, the paper confirms that the proposed method does not negatively affect non-numeral embeddings. And then, the quality of the numeral embeddings are evaluated and compared. The experiments show that the proposed method has better performance on numerical property tests, numeral prediction, and a sequence labeling task.