Word2Fun: Modelling Words as Functions for Diachronic Word Representation

1 Introduction . Word meaning changes over time as a reflection of the changes of human society . Not only does word meaning change in a short time , word meaning may also be subject to evolution over long timespans . The causes of the change of word meaning may be cultural , societal , or technological [ 23 ] . Such lexical semantic change phenomenon has been investigated for some decades [ 3 , 23 ] . Modeling the change of word meaning recently relies on distributed representations of words [ 2 , 14 ] and adopting time-specific word vectors as diachronic word representation . The training is based on diachronic text corpora , which are obtained by partitioning the corpora into bins of textual documents labeled by time . A popular training paradigm within such an approach is known as train-and-align : first , the embeddings are trained separately for each pre-grouped time-stamped corpus ; then , the embeddings are aligned by computing , e.g . orthogonal projections [ 9 , 12 ] , vector initialization [ 11 ] , temporal referencing [ 7 ] , parameter regularizers [ 26 , 20 ] , aligned compass [ 4 ] , or latent diffusion [ 1 ] . The methods above do not model word meaning evolution as a continuous process , but consider one-hop transformation between pair of timestamps . However , the meaning of some words may smoothly evolve across a long-range span , according to a mixture of short and long timespans or in general according to unknown patterns which can not straightforwardly be represented only considering one-hop transformations . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . A recent approach called ‘ DiffTime ’ [ 18 ] models semantic evolution as functions parameterized by multiple-layer nonlinear neural networks over time . Our paper carries on this line of work by modeling the change of word meaning as implicit functions for diachronic word representation , and considering word meaning evolution as a continuous process . These functions have considerable expressive power since polynomials can approximate any functions with a given negligible error thanks to the Weierstrass Approximation Theorem . Even if the proposed approach is general , we will focus on trigonometric functions . We experimentally show that modelling each word as a finite number of trigonometric functions combined together in trigonometric polynomials is an effective approach to modeling the change of word meaning . Our experimental investigation also shows that the proposed method , even using fewer parameters , achieves better performance than the state-of-the-art ( SOTA ) diachronic word representation in time-aware word clustering , temporal analogy , and semantic change detection tasks , showing its potential for the research in diachronic word meaning representation and processing . 2 Background . 2.1 Problem Definition . Representing words in vector space , i.e. , a static word embedding [ 14 ] h : N→ RD , is a common practice . However , static word embedding can not model word dynamics over time , which is crucial when considering time-dependent corpora and some tasks , such as those carried out by specialists like linguists or sociologists . Diachronic word embedding ( DWE ) [ 9 ] ( sometimes called Dynamic Word Embedding [ 1 ] ) assumes that each word has a meaning which may vary with time . One word wi with index i has its meaning represented as a D-dimensional vector Ui , t ∈ RD at a given time t. Therefore , DWE can be formalized as a mapping from N× N to RD , i.e . from the pair ( i , t ) to a D-dimensional vector defined over a real field . Most methods for DWE separately train embeddings for each time t ∈ N. Note that even if we assume t ∈ N , all formalizations can be extended to t ∈ R ; in practice , the granularity between two consecutive times is fixed , e.g. , one year , thus representing t in N is enough . The vectors that represent V words in time t are defined as U· , t def = [ U1 , t , U2 , t · · · , UV , t ] ∈ RV D . 2.2 Alignment Issues for Rotation-invariant Embeddings . There are typically two ways to obtain word vectors : one method is matrix factorization , the other method is prediction-based neural methods ; the former method is also related to the latter [ 13 ] . When considering the matrix factorization case , a common approach [ 26 ] is to approximate a given Positive Pointwise Mutual Information ( PPMI ) matrix Mt ∈ RV×V by means of the V word vectors U· , t such that U· , tUT· , t ≈ Mt . The entries of Mt are measures of a relationship between words ; for example , the entry at row i and column j may measure the degree of co-occurrence of the i-th word with the j-word within fixed-size textual windows . The word embeddings can be obtained by either using an eigenvalue method or a matrix factorization method which finds U· , t = arg min V |V V T −Mt|F ( 1 ) However , we have that U′· , t = U· , tR can be an alternative solution of Eq . 1 for any orthogonal rotation matrix R ∈ RD×D . Indeed , U· , tR ( U· , tR ) T = U· , tRRT ( U· , t ) T = U· , t ( U· , t ) T , since RRT = I . Therefore , training word embeddings according to Eq . 1 is non-convex [ 1 ] and as a consequence a global minimum might not exist . Moreover , if diachronic word embeddings are trained twice on the same time-specific corpus yet with different random seeds , different solutions may result . It is also possible that two word vectors corresponding to different times , i.e. , U· , t1 and U· , t2 can not be compared because they might be arbitrarily rotated . One has to align these time-specific word embeddings . The rotation-invariant issue is also present for prediction-based word embeddings e.g. , Word2Vec [ 14 ] . 2.3 Existing Work of Diachronic Word Embedding . Existing DWE , e.g. , [ 1 , 9 ] considers word meaning evolution in two consecutive time-specific corpora ( e.g. , t1 and t2 ) , instead of a whole continuous process . They train word vectors for each time-specific corpus and then align them using orthogonal projections [ 9 , 12 ] , vector initialization [ 11 ] , temporal referencing [ 7 ] , parameter regularizers [ 26 , 20 ] , aligned compass [ 4 ] , latent diffusion [ 1 ] . Tab . 1 summarizes these approaches . The number of parameters of the alignment-based dynamic word embeddings is proportional to the number T of time bins , resulting in a parameter space of O ( TV D ) . This is a notable cost with a long time span , e.g. , splitting COHA dataset yearly results in 200 time slices ( T = 200 ) . Furthermore , transformations between two any consecutive time bins are independent , namely , Ui , t1 , Ui , t2 may be totally independent of the one between Ui , t2 , Ui , t3 even if they are close in the time . Note that these approaches might lose some power when modeling such sequential evolution with more than two time bins in the event of word meaning that gradually evolves during a longer timespan , e.g. , many decades . Lastly , dividing a corpus into separate time bins may lead to many smaller training sets . Word vectors in a time bin are trained with roughly 1T training examples instead of all examples . This might be an issue with time-unbalanced dynamic corpora . For example , when there are much fewer documents in a specific year t than others , the word-specific word vectors in that year may be distorted because of underfitting ; it may therefore negatively affect the long-range transformation of word meaning change . DiffTime [ 18 ] was the first work to model word meaning evolution as a continuous process to the best of our knowledge . It leverages multiple-layer neural networks to aggregate output from a word encoder f and a time encoder g. Even if the parameter scale does not increase with a bigger T , DiffTime additionally introduces much more parameters ( e.g. , D3 ) . Such learned functions are less interpretable due to the depth and non-linearity of neural networks . We argue that the neural network architecture in [ 18 ] is ad hoc without theoretical approximation guarantee for semantic shift . 3 Methodology : Modelling Words as Functions . 3.1 Formulation . To smoothly model word meaning change over time we represent each word as a continuous function : a specific word vector at time t is represented as the values of the function when the variable equals t , which is inspired by [ 24 , 18 , 10 , 25 ] . More formally , our approach aims to learn a mapping f that maps each word wi , with index i ∈ N , to functions over time : f : N→ ( g : N→ RD ) where g is a function over a variable t ∈ N. Note that the output of g is a D-dimensional vector , g ( t ) ∈ RD . Let us denote f ( i ) as gi . A word wi at time t is represented as a D-dimensional vector Ui , t = f ( i ) ( t ) = gi ( t ) . Examples of g are linear functions g ( t ) = b+kt with parameters b , k ∈ RD or a sinusoidal functions g ( t ) = b+ r sin ( ωt+ θ ) with parameters b , r , ω , θ ∈ RD . Section 4 reports some remarks on the considered instantiations of g and how they are linked to previously proposed approaches . 3.2 A Skip-gram Implementation with a Compass . The Skip-gram model architecture tries to predict the source context words ( surrounding words ) given a target word ( the center word ) , which was considered to achieve the reverse of what the CBOW model does . Let us denote u as the target word and v as the source word , and negatively-sampled target words V̂ = { v̂i } . It learns a diachronic word representation method f : N× R→ RD and a static word mapping h : N→ RD ; f ( u , t ) denotes the diachronic word vector of wu in time t. The static word embedding h ( u ) is introduced to align the diachronic word vectors in different times ; it is called a compass in [ 4 ] . The compass is used as an anchor to project time-specific word embedding in different times to unified designated vector space . Our implementation of the compass differs from [ 4 ] , since we jointly train the static compass and diachronic word vectors simultaneously , thus avoiding a two-stage training and the compass can be non-fixed in the second stage . The loss function is defined as : L = − ∑ ( u , v , V̂ , t ) ∈Dtrain log ( δ ( f ( u , t ) h ( v ) T ) ) + 1 |V̂| ∑ v̂i∈V̂ log ( δ ( −f ( u , t ) h ( v̂i ) T ) ) ( 2 ) δ is the sigmoid activation function . The algorithm is explained in Algo . 1 , and the details on the parameterization are reported in Tab . 1 . ALGORITHM 1 : Training algorithm for Word2Fun . Require : time-stamped corpora C ; a diachronic word mapping f : ( N , R ) → ND a static word mapping h : N→ RD 1 : shuffling time-stamped corpora 2 : for each epoch do 3 : for each document d with a timestamp t in C do 4 : for each skip-gram ( u , v ) in d do 5 : calculating the loss for the positive sample Lpos = log ( δ ( f ( u , t ) h ( v ) T ) ) 6 : generating negatively-sampled target words V̂ 7 : calculating the loss for negative samples Lneg = − ∑ v̂i∈V̂ log ( δ ( −f ( u , t ) h ( v̂i ) T ) ) 8 : back-propagating using the loss L = Lpos + Lneg 9 : end for 10 : end for 11 : end for

Summary:  This paper presents a new approach to modeling word embedding change over time: each word $w_i$ is represented as a function $g_i(t) : N \rightarrow R^D$ that maps a timestamp $t \in \mathbb{N}$ to an embedding $e_{it} \in \mathbb{R}^D$. The function $g_i(t)$ can then be parameterized in a variety of ways; the paper focuses primarily on sinusoidal parameterizations. The paper motivates this approach via the Weierstrass theorem, from which it follows that a real function on the interval [a,b] can be well approximated by a trigonometric polynomial.  The key contribution of this paper is that it breaks out of the typical paradigm that has been used for diacronic word embeddings: i.e., that a separate word embedding $e_{it}$ is explicitly learned for each word $w_i$ and each timestamp $t$. Doing so comes with several drawbacks that this paper addresses: (1) that an alignment between timestamps must be learned (2) that the alignments are (usually) only dependent in a pairwise Markovian fashion, thus missing out on longer time effects (3) that space required is high.  This paper evaluates its method with different parametric functions (e.g. linear, word/time independent, and various sinusoids) on several benchmarks established in prior work for the evaluation of diacrhonic word embeddings and attains (mostly) better results than strong baselines from prior work.