Learning Numeral Embedding

1 INTRODUCTION . Word embeddings , the distributed vector representations of words , have become the essential building block for deep learning approaches to natural language processing ( NLP ) . The quality of pretrained word embeddings has been shown to significantly impact the performance of neural approaches to a variety of NLP tasks . Over the past two decades , significant progress has been made in the development of word embedding techniques ( Lund & Burgess , 1996 ; Bengio et al. , 2003 ; Bullinaria & Levy , 2007 ; Mikolov et al. , 2013b ; Pennington et al. , 2014 ) . However , existing word embedding methods do not handle numerals adequately and can not directly encode the numeracy and magnitude of a numeral ( Naik et al. , 2019 ) . Most methods have a limited vocabulary size and therefore can only represent a small subset of the infinite number of numerals . Furthermore , most numerals have very scarce appearances in training corpora and therefore are more likely to be outof-vocabulary ( OOV ) compared to non-numerical words . For example , numerals account for 6.15 % of all unique tokens in English Wikipedia , but in GloVe Pennington et al . ( 2014 ) which is partially trained on Wikipedia , only 3.79 % of its vocabulary is numerals . Previous work ( Spithourakis et al. , 2016 ) also shows that the numeral OOV problem is even more severe when learning word embeddings from corpora with abundant numerals such as clinical reports . Even if a numeral is included in the vocabulary , its scarcity in the training corpus would negatively impact the learning accuracy of its embedding . The inadequate handling of numerals in existing word embedding methods can be problematic in scenarios where numerals convey critical information . Take the following sentences for example , “ Jeff is 190 , so he should wear size XXL ” ( 190 is a reasonable height for size XXL ) “ Jeff is 160 , so he should wear size XXL ” ( 160 is an unreasonable height for size XXL ) “ Jeff is 10 , so he should wear size XS ” ( 10 is an age instead of a height ) If the numerals in the example are OOV or their embeddings are not accurately learned , then it becomes impossible to judge the categories of the numerals or the reasonableness of the sentences . In this paper , we propose two novel methods that can produce reasonable embeddings for any numerals . The key idea is to represent the embedding of a numeral as a weighted average of a small set of prototype number embeddings . The prototype numerals are induced from the training corpus using either a self-organizing map ( Kohonen , 1990 ) or a Gaussian mixture model . The weights are computed based on the differences between the target numeral and the prototype numerals , reflecting the inductive bias that numerals with similar quantities are likely to convey similar semantic information and thus should have similar embeddings . Numeral embeddings represented in this manner can then be plugged into a traditional word embedding method for training . We empirically evaluate our methods on four tasks : word similarity , embedding numeracy , numeral prediction , and sequence labeling . The results show that our methods can produce high-quality embeddings for both numerals and non-numerical words and improve the performance of downstream tasks . 2 RELATED WORK . Word Embedding Word embeddings are vector representations of words which carry semantic meanings implicitly and are trained without supervision . Most existing word embedding training methods can be divided into two classes . The first class of methods ( Lund & Burgess , 1996 ; Rohde et al. , 2006 ; Bullinaria & Levy , 2007 ; Lebret & Lebret , 2013 ) extract word co-occurrence statistics from the training corpus , compute a word-word matrix based on measures such as PPMI , and then apply dimension reduction techniques such as principle component analysis to produce a low-dimensional vector representation for each word . The second class of methods ( Bengio et al. , 2003 ; Collobert & Weston , 2008 ; Mikolov et al. , 2013a ; b ) use a simple neural network to model the relation between a word and its context within a sliding window in the training corpus . GloVe ( Pennington et al. , 2014 ) has been proposed as a method that combines the advantages of both classes . All the above methods have a finite vocabulary size and use a ‘ UNK ’ symbol to represent OOV words . Recent work ( Naik et al. , 2019 ) shows that these popular methods do not handle numerals adequately . Wallace et al . ( 2019 ) shows that existing word embedding methods can encode numeracy implicitly for high-frequency numerals , but the embedding ’ s numeracy for OOV numerals is not investigated . Our goal is to design better numeral embedding methods that can be integrated into traditional word embedding methods and handle the OOV problem for numerals . Numeracy in natural language Numeral understanding has been found important in textual entailment ( Lev et al. , 2004 ; De Marneffe et al. , 2008 ; Roy et al. , 2015 ) and information extraction ( Intxaurrondo et al. , 2015 ; Madaan et al. , 2016 ) , but existing systems often use manually defined task-specific features and logic rules to identify numerals , which is hard to generalize to other tasks . A lot of research has been done trying to solve math problems , using either manually designed features and rules ( Roy et al. , 2015 ; Mitra & Baral , 2016 ; Roy & Roth , 2016 ; Upadhyay et al. , 2016 ) or sequence-to-sequence neural networks Wang et al . ( 2017 ) , but the quantity of numerals is not important in this task and hence existing methods often replace numerals by dummy symbols such as n1 and n2 . Spithourakis & Riedel ( 2018 ) studied different strategies to better model numerals in language models . Chen et al . ( 2019 ) created Numeracy-600K dataset and studied the ability of neural network models to learn numeracy . Our work differs from previous work in that we aim to produce general-purpose numeral embeddings that can be employed in any neural NLP approach . 3 METHODS . Given a training corpus C , we first extract all the numerals using regular expressions and form a dataset X containing all the numbers represented by these numerals . A number ( e.g. , 2000 ) may appear for multiple times in X if its corresponding numerals ( e.g. , ‘ 2000 ’ , ‘ 2,000 ’ , etc . ) appear for multiple times in C. We then induce a finite set P of typical numerals ( i.e. , prototypes ) from X using a self-organizing map ( Kohonen , 1990 ) or a Gaussian mixture model . We also define a function sim ( n1 , n2 ) outputting the similarity between two arbitrary numbers n1 and n2 . Now we represent the embedding of any target numeral n as a weighted average of the prototype number embeddings with the weights computed by the similarity function : e ( n ) = · ∑ p∈P α · sim ( n , p ) · e ( p ) , ∑ p∈P α · sim ( n , p ) = 1 ( 1 ) We use e ( · ) to denote the embedding of a number α is the normalization factor . This formulation satisfies the intuition that numerals with similar quantities are likely to convey similar semantic information and thus should have similar embeddings . Our numeral embeddings can be integrated into traditional word embedding methods such as skipgram for training . During training , we back-propagate the error gradient to update the prototype number embeddings . In this way , the prototype number embeddings ( and hence all the numeral embeddings ) are learned jointly with non-numerical word embeddings . 3.1 SQUASHING NUMBERS TO LOG-SPACE . Inspired by psychological evidence that our brain compresses large quantities nonlinearly using a logarithmic scale on the mental number line ( Nieder & Miller , 2003 ; Dehaene , 2011 ) , we design the following squashing function to transform all the numbers in X into the log-space before prototype induction . Alternatively , we can apply the function only in the similarity function . Besides the psychological motivation , squashing is also necessary for our methods to avoid overflow during training when there are very large numbers such as 1015 in the training corpus . f ( x ) = { log ( x ) + 1 , if x > 1 x , if x ∈ [ −1 , 1 ] − log ( −x ) − 1 , if x < −1 ( 2 ) 3.2 PROTOTYPE INDUCTION . We develop two methods for inducing a small prototype set P from the number dataset X . Denote the number of prototypes by m. Self-Organizing Map A self-organizing map ( SOM ) ( Kohonen , 1990 ) is an artificial neural network that can be viewed as a clustering method . After training a SOM on the dataset X , we regard each cluster centroid as a prototype . One advantage of using a SOM in comparison with traditional clustering methods is that it distributes prototypes more evenly on the number line and may assign prototypes to number ranges with few training samples , which we expect would lead to better generalizability . Gaussian Mixture Model Inspired by psychological study of the mental number line ( Dehaene et al. , 2003 ) and previous work on language modeling ( Spithourakis & Riedel , 2018 ) , we train a Gaussian mixture model ( GMM ) to induce number prototypes . A GMM is defined as follows . p ( U = n ) = m∑ k=1 P ( Z = k ) P ( U = n|Z = k ) = m∑ k=1 πkN ( n ; µk , σ2k ) ( 3 ) where Z is a latent variable representing the mixture component for random variable U , and N is the probability density function of a normal distribution , and πk , µk , σk ∈ R represent the mixing coefficient , mean and standard deviation of the k-th Gaussian component . We train a GMM on the number dataset X using the expectation-maximization ( EM ) or hard-EM algorithm and regard the means of the learned Gaussian components as our prototypes P = { µ1 , · · · , µm } . We use three GMM initialization methods described in Appendix A . 3.3 SIMILARITY FUNCTION . For SOM-induced prototypes , we define the following similarity function : sim ( p , n ) = |g ( p ) − g ( n ) |−β , β > 0 , p ∈ P ( 4 ) where function g is equal to the squashing function f defined in Eq.2 if we do not apply log transformation before prototype induction and is the identity function I otherwise . β is a hyper-parameter set to 1.0 by default . For GMM-induced prototypes , we can naturally use the posterior probability of the component assignment to define the similarity function . sim ( pk , n ) ∝ P ( Z = k|U = n ) = πkN ( n ; µk , σ2k ) ∑m k=1 πkN ( n ; µk , σ2k ) , pk ∈ P ( 5 ) 3.4 EMBEDDING TRAINING . We now describe how to integrate our numeral embeddings into traditional word embedding methods for training . We choose skip-gram with negative sampling ( Mikolov et al. , 2013a ; b ) as the word embedding method here , but many other word embedding methods such as CBOW ( Mikolov et al. , 2013a ) , HAL ( Lund & Burgess , 1996 ) and GloVe ( Pennington et al. , 2014 ) can be used as well . Skip-gram is a word embedding method based on the idea of context word prediction . The training corpus C is regarded as a sequence of words ( x1 , . . . , xT ) . For token xt , we define the preceding and following c tokens as the context of xt . Skip-gram aims to maximize p ( xt+j |xt ) ( −c ≤ j ≤ c ) , the probability of a context word given the center word xt . To formulate p ( xt+j |xt ) , skip-gram associates each word xi with two vector representations : the input embedding vixt for being a center word and the output embedding voxt for being a context word . The input and output embeddings of all the words in the vocabulary V constitute matrices EI ∈ RD×|V| and EO ∈ RD×|V| respectively , where D is the dimension of word embeddings . The conditional probability p ( xt+j |xt ) is then defined to based on the dot product s ( xt+j |xt ) = vixt T voxt+j . Nagative sampling is used to approximate the normalization factor for the conditional probability . log p ( xt+j |xt ) ≈ log σ ( voxt+j T vixt ) + k∑ i=1 E xi∼Pn ( x ) [ log σ ( −voxi T vixt ) ] ( 6 ) where σ denotes the sigmoid function , and Pn ( x ) is the negative word sampling distribution used to draw k negative samples . We modify skip-gram by computing numeral embeddings differently from non-numerical word embeddings . We associate each prototype number with an input embedding and an output embedding . The input and output embeddings of all the prototypes constitute matrices MI ∈ RD×|P| and MO ∈ RD×|P| respectively . For any numeral , we can compute its input and output embeddings by taking a weighted average of the prototype input and output embeddings respectively based on Eq.1 and use them in exactly the same way as the embeddings of non-numerical words to compute the learning objective ( Eq.6 ) . When drawing negative samples , we first set the ratio of numerals and non-numerical words to their actual ratio in the training corpus , to guarantee a sufficient number of numeral negative samples . Then we sample numerals and non-numerical words separately from their respective distributions in the training corpus raised to the power of 34 . During training , we optimize the objective function Eq.6 by back-propagating the gradient of error to update both the embedding matrices both the non-numerical word embedding matrices EI , EO and the prototype number embedding matrices MI , MO . In this way , the embeddings of non-numerical words and numerals are learned jointly in the same space . We show an example in Figure 1 . Methods WS353 MEN SIM999 SOM 64.40 71.79 36.09 GMM 64.90 71.89 36.29 NumAsTok 65.30 71.83 35.85 D-LSTM 63.60 71.82 34.58 Fixed 64.35 72.17 36.27 SG GoogleNews-100B 70.00 74.10 44.20 GloVe Wiki-6B 52.20 73.70 37.10 Table 2 : Results on word similarity tasks trained on Wiki-1B . For reference , we also show the results of the official skip-gram and GloVe trained on larger corpora .

Summary: The paper talks about a recently highlighted problem in word embeddings which is their incapability to represent numerals, especially the out-of-vocabulary numerals. For addressing the problem, they propose a method that induces a finite set of prototype numerals using either self-organizing map or Gaussian Mixture model. Then, each numeral is represented as a weighted average of prototype numeral embeddings. The method also involves squashing large quantities using log function. Finally, the training is performed similar to Skip-gram in word2vec but with the embedding of numerals computed using prototype numerals.