Learning Numeral Embedding

1 INTRODUCTION . Word embeddings , the distributed vector representations of words , have become the essential building block for deep learning approaches to natural language processing ( NLP ) . The quality of pretrained word embeddings has been shown to significantly impact the performance of neural approaches to a variety of NLP tasks . Over the past two decades , significant progress has been made in the development of word embedding techniques ( Lund & Burgess , 1996 ; Bengio et al. , 2003 ; Bullinaria & Levy , 2007 ; Mikolov et al. , 2013b ; Pennington et al. , 2014 ) . However , existing word embedding methods do not handle numerals adequately and can not directly encode the numeracy and magnitude of a numeral ( Naik et al. , 2019 ) . Most methods have a limited vocabulary size and therefore can only represent a small subset of the infinite number of numerals . Furthermore , most numerals have very scarce appearances in training corpora and therefore are more likely to be outof-vocabulary ( OOV ) compared to non-numerical words . For example , numerals account for 6.15 % of all unique tokens in English Wikipedia , but in GloVe Pennington et al . ( 2014 ) which is partially trained on Wikipedia , only 3.79 % of its vocabulary is numerals . Previous work ( Spithourakis et al. , 2016 ) also shows that the numeral OOV problem is even more severe when learning word embeddings from corpora with abundant numerals such as clinical reports . Even if a numeral is included in the vocabulary , its scarcity in the training corpus would negatively impact the learning accuracy of its embedding . The inadequate handling of numerals in existing word embedding methods can be problematic in scenarios where numerals convey critical information . Take the following sentences for example , “ Jeff is 190 , so he should wear size XXL ” ( 190 is a reasonable height for size XXL ) “ Jeff is 160 , so he should wear size XXL ” ( 160 is an unreasonable height for size XXL ) “ Jeff is 10 , so he should wear size XS ” ( 10 is an age instead of a height ) If the numerals in the example are OOV or their embeddings are not accurately learned , then it becomes impossible to judge the categories of the numerals or the reasonableness of the sentences . In this paper , we propose two novel methods that can produce reasonable embeddings for any numerals . The key idea is to represent the embedding of a numeral as a weighted average of a small set of prototype number embeddings . The prototype numerals are induced from the training corpus using either a self-organizing map ( Kohonen , 1990 ) or a Gaussian mixture model . The weights are computed based on the differences between the target numeral and the prototype numerals , reflecting the inductive bias that numerals with similar quantities are likely to convey similar semantic information and thus should have similar embeddings . Numeral embeddings represented in this manner can then be plugged into a traditional word embedding method for training . We empirically evaluate our methods on four tasks : word similarity , embedding numeracy , numeral prediction , and sequence labeling . The results show that our methods can produce high-quality embeddings for both numerals and non-numerical words and improve the performance of downstream tasks . 2 RELATED WORK . Word Embedding Word embeddings are vector representations of words which carry semantic meanings implicitly and are trained without supervision . Most existing word embedding training methods can be divided into two classes . The first class of methods ( Lund & Burgess , 1996 ; Rohde et al. , 2006 ; Bullinaria & Levy , 2007 ; Lebret & Lebret , 2013 ) extract word co-occurrence statistics from the training corpus , compute a word-word matrix based on measures such as PPMI , and then apply dimension reduction techniques such as principle component analysis to produce a low-dimensional vector representation for each word . The second class of methods ( Bengio et al. , 2003 ; Collobert & Weston , 2008 ; Mikolov et al. , 2013a ; b ) use a simple neural network to model the relation between a word and its context within a sliding window in the training corpus . GloVe ( Pennington et al. , 2014 ) has been proposed as a method that combines the advantages of both classes . All the above methods have a finite vocabulary size and use a ‘ UNK ’ symbol to represent OOV words . Recent work ( Naik et al. , 2019 ) shows that these popular methods do not handle numerals adequately . Wallace et al . ( 2019 ) shows that existing word embedding methods can encode numeracy implicitly for high-frequency numerals , but the embedding ’ s numeracy for OOV numerals is not investigated . Our goal is to design better numeral embedding methods that can be integrated into traditional word embedding methods and handle the OOV problem for numerals . Numeracy in natural language Numeral understanding has been found important in textual entailment ( Lev et al. , 2004 ; De Marneffe et al. , 2008 ; Roy et al. , 2015 ) and information extraction ( Intxaurrondo et al. , 2015 ; Madaan et al. , 2016 ) , but existing systems often use manually defined task-specific features and logic rules to identify numerals , which is hard to generalize to other tasks . A lot of research has been done trying to solve math problems , using either manually designed features and rules ( Roy et al. , 2015 ; Mitra & Baral , 2016 ; Roy & Roth , 2016 ; Upadhyay et al. , 2016 ) or sequence-to-sequence neural networks Wang et al . ( 2017 ) , but the quantity of numerals is not important in this task and hence existing methods often replace numerals by dummy symbols such as n1 and n2 . Spithourakis & Riedel ( 2018 ) studied different strategies to better model numerals in language models . Chen et al . ( 2019 ) created Numeracy-600K dataset and studied the ability of neural network models to learn numeracy . Our work differs from previous work in that we aim to produce general-purpose numeral embeddings that can be employed in any neural NLP approach . 3 METHODS . Given a training corpus C , we first extract all the numerals using regular expressions and form a dataset X containing all the numbers represented by these numerals . A number ( e.g. , 2000 ) may appear for multiple times in X if its corresponding numerals ( e.g. , ‘ 2000 ’ , ‘ 2,000 ’ , etc . ) appear for multiple times in C. We then induce a finite set P of typical numerals ( i.e. , prototypes ) from X using a self-organizing map ( Kohonen , 1990 ) or a Gaussian mixture model . We also define a function sim ( n1 , n2 ) outputting the similarity between two arbitrary numbers n1 and n2 . Now we represent the embedding of any target numeral n as a weighted average of the prototype number embeddings with the weights computed by the similarity function : e ( n ) = · ∑ p∈P α · sim ( n , p ) · e ( p ) , ∑ p∈P α · sim ( n , p ) = 1 ( 1 ) We use e ( · ) to denote the embedding of a number α is the normalization factor . This formulation satisfies the intuition that numerals with similar quantities are likely to convey similar semantic information and thus should have similar embeddings . Our numeral embeddings can be integrated into traditional word embedding methods such as skipgram for training . During training , we back-propagate the error gradient to update the prototype number embeddings . In this way , the prototype number embeddings ( and hence all the numeral embeddings ) are learned jointly with non-numerical word embeddings . 3.1 SQUASHING NUMBERS TO LOG-SPACE . Inspired by psychological evidence that our brain compresses large quantities nonlinearly using a logarithmic scale on the mental number line ( Nieder & Miller , 2003 ; Dehaene , 2011 ) , we design the following squashing function to transform all the numbers in X into the log-space before prototype induction . Alternatively , we can apply the function only in the similarity function . Besides the psychological motivation , squashing is also necessary for our methods to avoid overflow during training when there are very large numbers such as 1015 in the training corpus . f ( x ) = { log ( x ) + 1 , if x > 1 x , if x ∈ [ −1 , 1 ] − log ( −x ) − 1 , if x < −1 ( 2 ) 3.2 PROTOTYPE INDUCTION . We develop two methods for inducing a small prototype set P from the number dataset X . Denote the number of prototypes by m. Self-Organizing Map A self-organizing map ( SOM ) ( Kohonen , 1990 ) is an artificial neural network that can be viewed as a clustering method . After training a SOM on the dataset X , we regard each cluster centroid as a prototype . One advantage of using a SOM in comparison with traditional clustering methods is that it distributes prototypes more evenly on the number line and may assign prototypes to number ranges with few training samples , which we expect would lead to better generalizability . Gaussian Mixture Model Inspired by psychological study of the mental number line ( Dehaene et al. , 2003 ) and previous work on language modeling ( Spithourakis & Riedel , 2018 ) , we train a Gaussian mixture model ( GMM ) to induce number prototypes . A GMM is defined as follows . p ( U = n ) = m∑ k=1 P ( Z = k ) P ( U = n|Z = k ) = m∑ k=1 πkN ( n ; µk , σ2k ) ( 3 ) where Z is a latent variable representing the mixture component for random variable U , and N is the probability density function of a normal distribution , and πk , µk , σk ∈ R represent the mixing coefficient , mean and standard deviation of the k-th Gaussian component . We train a GMM on the number dataset X using the expectation-maximization ( EM ) or hard-EM algorithm and regard the means of the learned Gaussian components as our prototypes P = { µ1 , · · · , µm } . We use three GMM initialization methods described in Appendix A . 3.3 SIMILARITY FUNCTION . For SOM-induced prototypes , we define the following similarity function : sim ( p , n ) = |g ( p ) − g ( n ) |−β , β > 0 , p ∈ P ( 4 ) where function g is equal to the squashing function f defined in Eq.2 if we do not apply log transformation before prototype induction and is the identity function I otherwise . β is a hyper-parameter set to 1.0 by default . For GMM-induced prototypes , we can naturally use the posterior probability of the component assignment to define the similarity function . sim ( pk , n ) ∝ P ( Z = k|U = n ) = πkN ( n ; µk , σ2k ) ∑m k=1 πkN ( n ; µk , σ2k ) , pk ∈ P ( 5 ) 3.4 EMBEDDING TRAINING . We now describe how to integrate our numeral embeddings into traditional word embedding methods for training . We choose skip-gram with negative sampling ( Mikolov et al. , 2013a ; b ) as the word embedding method here , but many other word embedding methods such as CBOW ( Mikolov et al. , 2013a ) , HAL ( Lund & Burgess , 1996 ) and GloVe ( Pennington et al. , 2014 ) can be used as well . Skip-gram is a word embedding method based on the idea of context word prediction . The training corpus C is regarded as a sequence of words ( x1 , . . . , xT ) . For token xt , we define the preceding and following c tokens as the context of xt . Skip-gram aims to maximize p ( xt+j |xt ) ( −c ≤ j ≤ c ) , the probability of a context word given the center word xt . To formulate p ( xt+j |xt ) , skip-gram associates each word xi with two vector representations : the input embedding vixt for being a center word and the output embedding voxt for being a context word . The input and output embeddings of all the words in the vocabulary V constitute matrices EI ∈ RD×|V| and EO ∈ RD×|V| respectively , where D is the dimension of word embeddings . The conditional probability p ( xt+j |xt ) is then defined to based on the dot product s ( xt+j |xt ) = vixt T voxt+j . Nagative sampling is used to approximate the normalization factor for the conditional probability . log p ( xt+j |xt ) ≈ log σ ( voxt+j T vixt ) + k∑ i=1 E xi∼Pn ( x ) [ log σ ( −voxi T vixt ) ] ( 6 ) where σ denotes the sigmoid function , and Pn ( x ) is the negative word sampling distribution used to draw k negative samples . We modify skip-gram by computing numeral embeddings differently from non-numerical word embeddings . We associate each prototype number with an input embedding and an output embedding . The input and output embeddings of all the prototypes constitute matrices MI ∈ RD×|P| and MO ∈ RD×|P| respectively . For any numeral , we can compute its input and output embeddings by taking a weighted average of the prototype input and output embeddings respectively based on Eq.1 and use them in exactly the same way as the embeddings of non-numerical words to compute the learning objective ( Eq.6 ) . When drawing negative samples , we first set the ratio of numerals and non-numerical words to their actual ratio in the training corpus , to guarantee a sufficient number of numeral negative samples . Then we sample numerals and non-numerical words separately from their respective distributions in the training corpus raised to the power of 34 . During training , we optimize the objective function Eq.6 by back-propagating the gradient of error to update both the embedding matrices both the non-numerical word embedding matrices EI , EO and the prototype number embedding matrices MI , MO . In this way , the embeddings of non-numerical words and numerals are learned jointly in the same space . We show an example in Figure 1 . Methods WS353 MEN SIM999 SOM 64.40 71.79 36.09 GMM 64.90 71.89 36.29 NumAsTok 65.30 71.83 35.85 D-LSTM 63.60 71.82 34.58 Fixed 64.35 72.17 36.27 SG GoogleNews-100B 70.00 74.10 44.20 GloVe Wiki-6B 52.20 73.70 37.10 Table 2 : Results on word similarity tasks trained on Wiki-1B . For reference , we also show the results of the official skip-gram and GloVe trained on larger corpora .

Summary: The paper proposes a novel method for embedding numerals which can be learned by using neural word embedding learning techniques. The paper motivates the work by reviewing the difficulty of embedding components to represent numerals: OOV in most cases. Their main contribution is the introduction of a method composes numeral embedding by a weighted average of prototype embeddings based on the similarities between the numeral and prototypes. There are two proposed prototypes: SOM and GMM and the similarity functions are an absolute difference and the density function respectively. During the training, the numerals have the proposed embeddings while the others have normal word embeddings. The paper slightly modifies the negative sampling to ensure numerals being sampled. A series of 4 empirical studies have been presented. First, the paper confirms that the proposed method does not negatively affect non-numeral embeddings. And then, the quality of the numeral embeddings are evaluated and compared. The experiments show that the proposed method has better performance on numerical property tests, numeral prediction, and a sequence labeling task.