Word2Fun: Modelling Words as Functions for Diachronic Word Representation

1 Introduction . Word meaning changes over time as a reflection of the changes of human society . Not only does word meaning change in a short time , word meaning may also be subject to evolution over long timespans . The causes of the change of word meaning may be cultural , societal , or technological [ 23 ] . Such lexical semantic change phenomenon has been investigated for some decades [ 3 , 23 ] . Modeling the change of word meaning recently relies on distributed representations of words [ 2 , 14 ] and adopting time-specific word vectors as diachronic word representation . The training is based on diachronic text corpora , which are obtained by partitioning the corpora into bins of textual documents labeled by time . A popular training paradigm within such an approach is known as train-and-align : first , the embeddings are trained separately for each pre-grouped time-stamped corpus ; then , the embeddings are aligned by computing , e.g . orthogonal projections [ 9 , 12 ] , vector initialization [ 11 ] , temporal referencing [ 7 ] , parameter regularizers [ 26 , 20 ] , aligned compass [ 4 ] , or latent diffusion [ 1 ] . The methods above do not model word meaning evolution as a continuous process , but consider one-hop transformation between pair of timestamps . However , the meaning of some words may smoothly evolve across a long-range span , according to a mixture of short and long timespans or in general according to unknown patterns which can not straightforwardly be represented only considering one-hop transformations . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . A recent approach called ‘ DiffTime ’ [ 18 ] models semantic evolution as functions parameterized by multiple-layer nonlinear neural networks over time . Our paper carries on this line of work by modeling the change of word meaning as implicit functions for diachronic word representation , and considering word meaning evolution as a continuous process . These functions have considerable expressive power since polynomials can approximate any functions with a given negligible error thanks to the Weierstrass Approximation Theorem . Even if the proposed approach is general , we will focus on trigonometric functions . We experimentally show that modelling each word as a finite number of trigonometric functions combined together in trigonometric polynomials is an effective approach to modeling the change of word meaning . Our experimental investigation also shows that the proposed method , even using fewer parameters , achieves better performance than the state-of-the-art ( SOTA ) diachronic word representation in time-aware word clustering , temporal analogy , and semantic change detection tasks , showing its potential for the research in diachronic word meaning representation and processing . 2 Background . 2.1 Problem Definition . Representing words in vector space , i.e. , a static word embedding [ 14 ] h : N→ RD , is a common practice . However , static word embedding can not model word dynamics over time , which is crucial when considering time-dependent corpora and some tasks , such as those carried out by specialists like linguists or sociologists . Diachronic word embedding ( DWE ) [ 9 ] ( sometimes called Dynamic Word Embedding [ 1 ] ) assumes that each word has a meaning which may vary with time . One word wi with index i has its meaning represented as a D-dimensional vector Ui , t ∈ RD at a given time t. Therefore , DWE can be formalized as a mapping from N× N to RD , i.e . from the pair ( i , t ) to a D-dimensional vector defined over a real field . Most methods for DWE separately train embeddings for each time t ∈ N. Note that even if we assume t ∈ N , all formalizations can be extended to t ∈ R ; in practice , the granularity between two consecutive times is fixed , e.g. , one year , thus representing t in N is enough . The vectors that represent V words in time t are defined as U· , t def = [ U1 , t , U2 , t · · · , UV , t ] ∈ RV D . 2.2 Alignment Issues for Rotation-invariant Embeddings . There are typically two ways to obtain word vectors : one method is matrix factorization , the other method is prediction-based neural methods ; the former method is also related to the latter [ 13 ] . When considering the matrix factorization case , a common approach [ 26 ] is to approximate a given Positive Pointwise Mutual Information ( PPMI ) matrix Mt ∈ RV×V by means of the V word vectors U· , t such that U· , tUT· , t ≈ Mt . The entries of Mt are measures of a relationship between words ; for example , the entry at row i and column j may measure the degree of co-occurrence of the i-th word with the j-word within fixed-size textual windows . The word embeddings can be obtained by either using an eigenvalue method or a matrix factorization method which finds U· , t = arg min V |V V T −Mt|F ( 1 ) However , we have that U′· , t = U· , tR can be an alternative solution of Eq . 1 for any orthogonal rotation matrix R ∈ RD×D . Indeed , U· , tR ( U· , tR ) T = U· , tRRT ( U· , t ) T = U· , t ( U· , t ) T , since RRT = I . Therefore , training word embeddings according to Eq . 1 is non-convex [ 1 ] and as a consequence a global minimum might not exist . Moreover , if diachronic word embeddings are trained twice on the same time-specific corpus yet with different random seeds , different solutions may result . It is also possible that two word vectors corresponding to different times , i.e. , U· , t1 and U· , t2 can not be compared because they might be arbitrarily rotated . One has to align these time-specific word embeddings . The rotation-invariant issue is also present for prediction-based word embeddings e.g. , Word2Vec [ 14 ] . 2.3 Existing Work of Diachronic Word Embedding . Existing DWE , e.g. , [ 1 , 9 ] considers word meaning evolution in two consecutive time-specific corpora ( e.g. , t1 and t2 ) , instead of a whole continuous process . They train word vectors for each time-specific corpus and then align them using orthogonal projections [ 9 , 12 ] , vector initialization [ 11 ] , temporal referencing [ 7 ] , parameter regularizers [ 26 , 20 ] , aligned compass [ 4 ] , latent diffusion [ 1 ] . Tab . 1 summarizes these approaches . The number of parameters of the alignment-based dynamic word embeddings is proportional to the number T of time bins , resulting in a parameter space of O ( TV D ) . This is a notable cost with a long time span , e.g. , splitting COHA dataset yearly results in 200 time slices ( T = 200 ) . Furthermore , transformations between two any consecutive time bins are independent , namely , Ui , t1 , Ui , t2 may be totally independent of the one between Ui , t2 , Ui , t3 even if they are close in the time . Note that these approaches might lose some power when modeling such sequential evolution with more than two time bins in the event of word meaning that gradually evolves during a longer timespan , e.g. , many decades . Lastly , dividing a corpus into separate time bins may lead to many smaller training sets . Word vectors in a time bin are trained with roughly 1T training examples instead of all examples . This might be an issue with time-unbalanced dynamic corpora . For example , when there are much fewer documents in a specific year t than others , the word-specific word vectors in that year may be distorted because of underfitting ; it may therefore negatively affect the long-range transformation of word meaning change . DiffTime [ 18 ] was the first work to model word meaning evolution as a continuous process to the best of our knowledge . It leverages multiple-layer neural networks to aggregate output from a word encoder f and a time encoder g. Even if the parameter scale does not increase with a bigger T , DiffTime additionally introduces much more parameters ( e.g. , D3 ) . Such learned functions are less interpretable due to the depth and non-linearity of neural networks . We argue that the neural network architecture in [ 18 ] is ad hoc without theoretical approximation guarantee for semantic shift . 3 Methodology : Modelling Words as Functions . 3.1 Formulation . To smoothly model word meaning change over time we represent each word as a continuous function : a specific word vector at time t is represented as the values of the function when the variable equals t , which is inspired by [ 24 , 18 , 10 , 25 ] . More formally , our approach aims to learn a mapping f that maps each word wi , with index i ∈ N , to functions over time : f : N→ ( g : N→ RD ) where g is a function over a variable t ∈ N. Note that the output of g is a D-dimensional vector , g ( t ) ∈ RD . Let us denote f ( i ) as gi . A word wi at time t is represented as a D-dimensional vector Ui , t = f ( i ) ( t ) = gi ( t ) . Examples of g are linear functions g ( t ) = b+kt with parameters b , k ∈ RD or a sinusoidal functions g ( t ) = b+ r sin ( ωt+ θ ) with parameters b , r , ω , θ ∈ RD . Section 4 reports some remarks on the considered instantiations of g and how they are linked to previously proposed approaches . 3.2 A Skip-gram Implementation with a Compass . The Skip-gram model architecture tries to predict the source context words ( surrounding words ) given a target word ( the center word ) , which was considered to achieve the reverse of what the CBOW model does . Let us denote u as the target word and v as the source word , and negatively-sampled target words V̂ = { v̂i } . It learns a diachronic word representation method f : N× R→ RD and a static word mapping h : N→ RD ; f ( u , t ) denotes the diachronic word vector of wu in time t. The static word embedding h ( u ) is introduced to align the diachronic word vectors in different times ; it is called a compass in [ 4 ] . The compass is used as an anchor to project time-specific word embedding in different times to unified designated vector space . Our implementation of the compass differs from [ 4 ] , since we jointly train the static compass and diachronic word vectors simultaneously , thus avoiding a two-stage training and the compass can be non-fixed in the second stage . The loss function is defined as : L = − ∑ ( u , v , V̂ , t ) ∈Dtrain log ( δ ( f ( u , t ) h ( v ) T ) ) + 1 |V̂| ∑ v̂i∈V̂ log ( δ ( −f ( u , t ) h ( v̂i ) T ) ) ( 2 ) δ is the sigmoid activation function . The algorithm is explained in Algo . 1 , and the details on the parameterization are reported in Tab . 1 . ALGORITHM 1 : Training algorithm for Word2Fun . Require : time-stamped corpora C ; a diachronic word mapping f : ( N , R ) → ND a static word mapping h : N→ RD 1 : shuffling time-stamped corpora 2 : for each epoch do 3 : for each document d with a timestamp t in C do 4 : for each skip-gram ( u , v ) in d do 5 : calculating the loss for the positive sample Lpos = log ( δ ( f ( u , t ) h ( v ) T ) ) 6 : generating negatively-sampled target words V̂ 7 : calculating the loss for negative samples Lneg = − ∑ v̂i∈V̂ log ( δ ( −f ( u , t ) h ( v̂i ) T ) ) 8 : back-propagating using the loss L = Lpos + Lneg 9 : end for 10 : end for 11 : end for

Summary: This paper proposes a new paradigm for modeling temporal word embeddings. Instead of the commonly prevalent approaches that train static embeddings for documents belonging to each timestamp followed by an alignment step, this work attempts to learn embeddings as a continuous function of time. This work proposes function parametrizations that use sinusoidal polynomials which can model arbitrary continuous functions. The approach is empirically compared against relevant baselines on time-aware clustering, temporal analogy, and semantic change detection.