Word2Fun: Modelling Words as Functions for Diachronic Word Representation
Year: 2022

Abstract:
Word meaning may change over time as a reflection of changes in human society. Therefore, modeling time in word representation is necessary for some diachronic tasks. Most existing diachronic word representation approaches train the embeddings separately for each pre-grouped time-stamped corpus and align these embeddings, e.g., by orthogonal projections, vector initialization, temporal referencing, and compass. However, not only does word meaning change in a short time, word meaning may also be subject to evolution over long timespans, thus resulting in a unified continuous process. A recent approach called ‘DiffTime’ models semantic evolution as functions parameterized by multiple-layer nonlinear neural networks over time. In this paper, we will carry on this line of work by learning explicit functions over time for each word. Our approach, called ‘Word2Fun’, reduces the space complexity from O(TV D) to O(kV D) where k is a small constant (k T ). In particular, a specific instance based on polynomial functions could provably approximate any function modeling word evolution with a given negligible error thanks to the Weierstrass Approximation Theorem. The effectiveness of the proposed approach is evaluated in diverse tasks including timeaware word clustering, temporal analogy, and semantic change detection. Code at: https://github.com/wabyking/Word2Fun.git.

heading: 1 Introduction
text: Word meaning changes over time as a reflection of the changes of human society. Not only does word meaning change in a short time, word meaning may also be subject to evolution over long timespans. The causes of the change of word meaning may be cultural, societal, or technological [23]. Such lexical semantic change phenomenon has been investigated for some decades [3, 23].
Modeling the change of word meaning recently relies on distributed representations of words [2, 14] and adopting time-specific word vectors as diachronic word representation. The training is based on diachronic text corpora, which are obtained by partitioning the corpora into bins of textual documents labeled by time. A popular training paradigm within such an approach is known as train-and-align: first, the embeddings are trained separately for each pre-grouped time-stamped corpus; then, the embeddings are aligned by computing, e.g. orthogonal projections [9, 12], vector initialization [11], temporal referencing [7], parameter regularizers [26, 20], aligned compass [4], or latent diffusion [1]. The methods above do not model word meaning evolution as a continuous process, but consider one-hop transformation between pair of timestamps. However, the meaning of some words may smoothly evolve across a long-range span, according to a mixture of short and long timespans or in general according to unknown patterns which cannot straightforwardly be represented only considering one-hop transformations.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
A recent approach called ‘DiffTime’ [18] models semantic evolution as functions parameterized by multiple-layer nonlinear neural networks over time. Our paper carries on this line of work by modeling the change of word meaning as implicit functions for diachronic word representation, and considering word meaning evolution as a continuous process. These functions have considerable expressive power since polynomials can approximate any functions with a given negligible error thanks to the Weierstrass Approximation Theorem. Even if the proposed approach is general, we will focus on trigonometric functions.
We experimentally show that modelling each word as a finite number of trigonometric functions combined together in trigonometric polynomials is an effective approach to modeling the change of word meaning. Our experimental investigation also shows that the proposed method, even using fewer parameters, achieves better performance than the state-of-the-art (SOTA) diachronic word representation in time-aware word clustering, temporal analogy, and semantic change detection tasks, showing its potential for the research in diachronic word meaning representation and processing.

heading: 2 Background

heading: 2.1 Problem Definition
text: Representing words in vector space, i.e., a static word embedding [14] h : N→ RD, is a common practice. However, static word embedding cannot model word dynamics over time, which is crucial when considering time-dependent corpora and some tasks, such as those carried out by specialists like linguists or sociologists. Diachronic word embedding (DWE) [9] (sometimes called Dynamic Word Embedding [1]) assumes that each word has a meaning which may vary with time. One word wi with index i has its meaning represented as a D-dimensional vector Ui,t ∈ RD at a given time t. Therefore, DWE can be formalized as a mapping from N× N to RD, i.e. from the pair (i, t) to a D-dimensional vector defined over a real field.
Most methods for DWE separately train embeddings for each time t ∈ N. Note that even if we assume t ∈ N, all formalizations can be extended to t ∈ R; in practice, the granularity between two consecutive times is fixed, e.g., one year, thus representing t in N is enough. The vectors that represent V words in time t are defined as U·,t def = [U1,t,U2,t · · · ,UV,t] ∈ RV D.

heading: 2.2 Alignment Issues for Rotation-invariant Embeddings
text: There are typically two ways to obtain word vectors: one method is matrix factorization, the other method is prediction-based neural methods; the former method is also related to the latter [13]. When considering the matrix factorization case, a common approach [26] is to approximate a given Positive Pointwise Mutual Information (PPMI) matrix Mt ∈ RV×V by means of the V word vectors U·,t such that U·,tUT·,t ≈ Mt. The entries of Mt are measures of a relationship between words; for example, the entry at row i and column j may measure the degree of co-occurrence of the i-th word with the j-word within fixed-size textual windows. The word embeddings can be obtained by either using an eigenvalue method or a matrix factorization method which finds
U·,t = arg min V |V V T −Mt|F (1)
However, we have that U′·,t = U·,tR can be an alternative solution of Eq. 1 for any orthogonal rotation matrix R ∈ RD×D. Indeed, U·,tR(U·,tR)T = U·,tRRT (U·,t)T = U·,t(U·,t)T , since RRT = I . Therefore, training word embeddings according to Eq. 1 is non-convex [1] and as a consequence a global minimum might not exist.
Moreover, if diachronic word embeddings are trained twice on the same time-specific corpus yet with different random seeds, different solutions may result. It is also possible that two word vectors corresponding to different times, i.e., U·,t1 and U·,t2 cannot be compared because they might be arbitrarily rotated. One has to align these time-specific word embeddings. The rotation-invariant issue is also present for prediction-based word embeddings e.g., Word2Vec [14].

heading: 2.3 Existing Work of Diachronic Word Embedding
text: Existing DWE, e.g., [1, 9] considers word meaning evolution in two consecutive time-specific corpora (e.g., t1 and t2), instead of a whole continuous process. They train word vectors for each time-specific corpus and then align them using orthogonal projections [9, 12], vector initialization [11], temporal referencing [7], parameter regularizers [26, 20], aligned compass [4], latent diffusion [1]. Tab. 1 summarizes these approaches. The number of parameters of the alignment-based dynamic word embeddings is proportional to the number T of time bins, resulting in a parameter space of O(TV D). This is a notable cost with a long time span, e.g., splitting COHA dataset yearly results in 200 time slices (T = 200). Furthermore, transformations between two any consecutive time bins are independent, namely, Ui,t1 ,Ui,t2 may be totally independent of the one between Ui,t2 ,Ui,t3 even if they are close in the time. Note that these approaches might lose some power when modeling such sequential evolution with more than two time bins in the event of word meaning that gradually evolves during a longer timespan, e.g., many decades. Lastly, dividing a corpus into separate time bins may lead to many smaller training sets. Word vectors in a time bin are trained with roughly 1T training examples instead of all examples. This might be an issue with time-unbalanced dynamic corpora. For example, when there are much fewer documents in a specific year t than others, the word-specific word vectors in that year may be distorted because of underfitting; it may therefore negatively affect the long-range transformation of word meaning change.
DiffTime [18] was the first work to model word meaning evolution as a continuous process to the best of our knowledge. It leverages multiple-layer neural networks to aggregate output from a word encoder f and a time encoder g. Even if the parameter scale does not increase with a bigger T , DiffTime additionally introduces much more parameters (e.g., D3). Such learned functions are less interpretable due to the depth and non-linearity of neural networks. We argue that the neural network architecture in [18] is ad hoc without theoretical approximation guarantee for semantic shift.

heading: 3 Methodology: Modelling Words as Functions

heading: 3.1 Formulation
text: To smoothly model word meaning change over time we represent each word as a continuous function: a specific word vector at time t is represented as the values of the function when the variable equals t, which is inspired by [24, 18, 10, 25]. More formally, our approach aims to learn a mapping f that maps each word wi, with index i ∈ N, to functions over time:
f : N→ (g : N→ RD)
where g is a function over a variable t ∈ N. Note that the output of g is a D-dimensional vector, g(t) ∈ RD. Let us denote f(i) as gi. A word wi at time t is represented as a D-dimensional vector Ui,t = f(i)(t) = gi(t).
Examples of g are linear functions g(t) = b+kt with parameters b,k ∈ RD or a sinusoidal functions g(t) = b+ r sin(ωt+ θ) with parameters b, r,ω,θ ∈ RD. Section 4 reports some remarks on the considered instantiations of g and how they are linked to previously proposed approaches.

heading: 3.2 A Skip-gram Implementation with a Compass
text: The Skip-gram model architecture tries to predict the source context words (surrounding words) given a target word (the center word), which was considered to achieve the reverse of what the CBOW model does. Let us denote u as the target word and v as the source word, and negatively-sampled target words V̂ = {v̂i}. It learns a diachronic word representation method f : N× R→ RD and a static word mapping h : N→ RD; f(u, t) denotes the diachronic word vector of wu in time t. The static word embedding h(u) is introduced to align the diachronic word vectors in different times; it is called a compass in [4]. The compass is used as an anchor to project time-specific word embedding in different times to unified designated vector space. Our implementation of the compass differs from [4], since we jointly train the static compass and diachronic word vectors simultaneously, thus avoiding a two-stage training and the compass can be non-fixed in the second stage.
The loss function is defined as:
L = − ∑
(u,v,V̂,t)∈Dtrain
log(δ(f(u, t)h(v)T )) + 1 |V̂| ∑ v̂i∈V̂ log(δ(−f(u, t)h(v̂i)T ))  (2) δ is the sigmoid activation function. The algorithm is explained in Algo. 1, and the details on the parameterization are reported in Tab. 1.
ALGORITHM 1: Training algorithm for Word2Fun. Require: time-stamped corpora C;
a diachronic word mapping f : (N,R)→ ND a static word mapping h : N→ RD
1: shuffling time-stamped corpora 2: for each epoch do 3: for each document d with a timestamp t in C do 4: for each skip-gram (u, v) in d do 5: calculating the loss for the positive sample Lpos = log(δ(f(u, t)h(v)T )) 6: generating negatively-sampled target words V̂ 7: calculating the loss for negative samples Lneg = − ∑ v̂i∈V̂ log(δ(−f(u, t)h(v̂i)
T )) 8: back-propagating using the loss L = Lpos + Lneg 9: end for
10: end for 11: end for

heading: 4 Which Functions?

heading: 4.1 Function Approximation using Polynomials
text: For a skip-gram pair (wi, wj),1 the similarity degree between a context word wi and a target word wj in time t is calculated as a dot product between the time-specific context embedding f(i, t) and the static target embedding h(j), the latter being the ‘compass’:
yi,j(t) = f(i, t)h(j) T (3)
yi,j(t) is function of time and it measures the similarity between wi and wj over time;2 we argue that semantic meaning evolution can be captured by approximating arbitrary between-word similarity over time. How to approximate yi,j(t) by selecting appropriate parameterization of f and h?
1If wi and wj appear together in a h-size window, we call (wi, wj) as a skip-gram pair in this paper, wi is the context word and wj is the target word, and vice versa.
2For PPMI factorization to obtain word vectors, yi,j(t) is the changing PPMI between wi and wj over time.
Let us recall the Weierstrass Approximation Theorem that states that every continuous real function defined on [a, b] can be approximated by a real polynomial. Note that Theorem 1 also holds for trigonometric polynomial defined in Definition 1, see Corollary 1.
Theorem 1. Weierstrass Approximation Theorem. Let F be a continuous real-valued function defined on the real interval [a, b]. For any > 0, there exists a polynomial P such that for all x ∈ [a, b], we have |F (x)− P (x)|∞ < . Definition 1. A trigonometric polynomial of degree D is an expression of the form ∆ +∑D k=1 αk cos(kx) + βk sin(kx), where ∆, α1, . . . , αD, β1, . . . , βD ∈ R.
Corollary 1. Approximation by trigonometric polynomials For every continuous 2π-periodical function F : R→ R defined on the real interval [0, 2π], and for any > 0, there exists a trigonometric polynomial P such that for all x ∈ [0, 2π], we have |F (x)− P (x)|∞ < . Corollary 2. The trigonometric polynomials span{1, sinx, cosx, · · · , sin(Dx), cos(Dx)} is dense in C[0, 2π] iff span{1, sin 2πxa−b , cos 2πx a−b , · · · , sin 2πDx a−b , cos 2πDx a−b } is dense in C[a, b].
From Corollary 1 [16, 27] we know that a trigonometric polynomial {0, sinx, cosx, · · · , sin(Dx), cos(Dx)} spans a subspace that can approximate any periodical continuous functions defined on [0, 2π], i.e., it is dense in C[0, 2π]. By Corollary 2, one can conclude that a weighted sum of trigonometric functions with appropriate periods can approximate any periodical continuous functions defined in an arbitrary closed interval [16].

heading: 4.2 Sinusoidal Parameterization in Word2Fun
text: Since the static embedding h(j) is not related to time t, we consider f(i, t) with sinusoidal paramerization (a mixture of cosine and sine functions plus a bias term). Then Eq. 3 will result in:3
yi,j(t) = f(i, t)h(j) T = ∑   Bi,1 + Ri,1 sin(Ω1t) Bi,2 + Ri,2 cos(Ω1t) · · · Bi,D−1 + Ri,D−1 sin(ΩD
2 t)
Bi,D + Ri,D cos(ΩD 2 t)
  Cj,1 Cj,2 · · ·
Cj,D−1 Cj,D
 
= D∑ k=1
Bi,kCj,k︸ ︷︷ ︸ ∆ +
D 2∑
k=1 Ri,2k−1Cj,2k−1︸ ︷︷ ︸ αi,j,k sin(Ωkt) + Ri,2kCj,2k︸ ︷︷ ︸ βi,j,k cos(Ωkt)
(4)
Therefore, yi,j(t) is a weighted sum of sinusoidal functions plus a constant term ∆ = ∑D k=1Bi,kCj,k. By replacing the coefficients with αi,j,k and βi,j,k, we can rewrite Eq. 4 as yi,j(t) = ∆ +∑D 2
k=1 αi,j,k sin(Ωkt) + βi,j,k cos(Ωkt); {αi,j,k} D 2 k=1 and {βi,j,k} D 2 k=1 are the coefficients and
{Ωk} D 2
k=1 are the corresponding frequencies. Following the argument in [5] when discussing approximation properties of sine and cosine activation functions, since linear combinations of sine and cosine generate all finite trigonometric polynomials that have approximation properties described in Section 4.1, the dot product between the diachronic embedding f(i, t) and the static target embedding h(i), denoted as yi,j(t) can capture any evolving relations between arbitrary skip-gram pairs.
By iteratively training all skip-gram pairs, Word2Fun can model complicated semantic change across time. Intuitively, small frequencies would reflect some long-range evolution, while some big frequencies would capture short-range evolution. Such periodical property would allow such functions to capture long enough evolution without considering boundedness issues.

heading: 4.3 Investigated Function Parameterizations
text: Linear functions. A real polynomial naturally induce: Ui,t = gi(t) = bi + ki [1, t, t2, · · · tD]T . Since tD is computationally expensive and sometimes meets the overflow issue, we choose a simpler case with a linear function parameterization of Word2fun: Ui,t = gi(t) = bi + kit.
3 is the element-wise multiplication
Sinusoidal functions. In this work, we investigate the sinusoidal parameterization. We use a combination of sine and cosine functions; i.e., half of dimensions are parameterized by sine functions and half of them are parameterized by cosine functions. Considering the domain of time t ∈ N and its range in this paper, we adopt a set of frequencies {ωi} [24] that is different from standard trigonometric polynomial. The Word2Fun II is defined as below:
gi(t) = bi + ri[cos(Ωt); sin(Ωt)], Ωj = 1
10000
j/D 2
, j ∈ [1, · · · , D 2 ] (5)
Note we add a static component bi that is independent from time. In Word2Fun III we also learn word-specific frequencies: Ui,t = gi(t) = bi + ri[sin(ω (1) i t); cos(ω (2) i t)]
4. Word2Fun IV adds trainable word-specific phases: Ui,t = gi(t) = bi + ri[sin(ω (1) i t + θ (1) i ); cos(ω (2) i t + θ (2) i )], see Tab. 1. ω(1)i ∈ R D 2 is different from ω(2)i ∈ R D 2 . θ(1)i ∈ R D 2 is different from θ(2)i ∈ R D 2 .
Standard static word vectors, e.g., [14], can be considered as constant functions, a special case of Word2fun since bi is a time-agnostic word vector. For instance, (1) in the case of the linear function, ki = 0; (2) in the case of sinusoidal function, ri = 0 or ωi is small enough. Both these cases results in gi = bi. The additional parameters ωi and ri are expected to capture the word meaning evolution.
Time2Fun: purely encoding time as functions. We can separately learn a word mapping and a time mapping: fword : N −→ RD, ftime : N −→ RD (6) Then wi at time t can be represented as Ui,t = fword(i) + ftime(t), where ftime can be a function over time, e.g., ftime = Kt or ftime = sin(Ωt) — the latter corresponds to Word2Fun I in Tab. 1. Time2Fun is close to [18] which represents time as a continuous variable and model a word’s usage as a function of time. [10] also explores to use time as an extra feature for various tasks.

heading: 5 Experiments

heading: 5.1 Data and Experimental Settings
text: Corpora. The diachronic corpora used in this paper are reported in Tab. 2. The Corpus of Historical American English (COHA) [6] is the largest structured corpus of historical English (the 1810s-2010s), contains more than 475 million words and is balanced by genre decade by decade. The New York Times (NYT) [26] contains 99,872 articles published between January 1990 and July 2016; besides the article text, metadata including title, author, release date, and section label were also collected.
Baselines. We choose the baselines from [26]. Static Word2Vec: the standard word2vec embeddings [14], trained on the entire corpus and ignoring time information. Transformed Word2Vec [12]: the embeddings are first trained separately by factorizing PPMI matrix for each year, and then transformed by optimizing a linear transformation matrix which minimizes the distance between two consequent time-stamped trained embeddings for the k nearest words’ embeddings to the querying words. Aligned Word2Vec [9] : the embeddings are first trained by factorizing the PPMI matrix for each year t, and then aligned by searching for the best orthonormal transformation between two consequent time-stamped trained embeddings. Dynamic Word Embedding [26] learns word embedding by factorizing PPMI matrix in individual time; plus, it imposes a regularizer to encourage two subsequent word embedding being similar for alignment. Besides those baselines, we considered DiffTime [18] (described in Sec. 2.3) and Compass Aligned Word Embedding [4] that exploits fixed target embeddings to align time-specific context embeddings. The former has a embedding size of 150 and its hidden dimension is 50, and the latter has a embedding size of 100; the both have more parameters than Word2Fun.
4Sinusoidal functions introduce infinite local minima due to the periodicity of sinusoidal functions [15], but we empirically found it performs well [10] and leave the above issue a future work.
Experimental settings. We removed words that appear less than 200 times, as [26] did. We used the Adam optimizer with a learning rate of 0.0025. The batch size can be as big as it achieves the upper bound of memory of GPU. The model converged in less than 20 epochs with no drop in terms of loss. We computed the average performance on the last three saved checkpoints. For sinusoidal functions, we used a mixture of cosine functions and sine functions. The dimension was 100; we preliminary found that models with bigger dimensions (e.g. 200 or 300) can slightly improve performance. Word2Fun with all settings was slightly slower than the standard Word2Vec due to additional computation for obtaining word vectors. However, all methods include Word2Vec and Word2Fun took 10-15 minutes for one epoch in a single Nvidia V100 32G GPU for NYT dataset.

heading: 5.2 Quantitative Evaluation

heading: 5.2.1 Time-aware Word Clustering
text: The time-aware word clustering task was proposed in [26] and exploits the NYT dataset. If one word was extremely frequent in a particular section, it was labeled as most-used in that section.5 Such section annotation was used to evaluate the clustering results. Across 11 sections, there were 1,888 triplets denoted as (wi, ti, si) to indicate that word wi in year ti was associated with section si.
We apply spherical K-means on learned time-specific word vectors using cosine distance, with K = 10, 15, and 20 clusters. As in [26], we used Normalized Mutual Information (NMI) and Fβmeasure to measure the consistency between section label and clustering results. NMI was defined as NMI(C,S) = 2MI(C,S)/(E(C) + E(S)), where S denotes section labels ({si}) for word-year pairs {(wi, ti)} and C denotes clustering categories ({ci}) for these word-year pairs {(wi, ti)}. MI(·, ·) and E(·) are the functions to compute Mutual Information and Entropy respectively. Fβ= (β 2+1)PR β2P+R measures the effectiveness as a β-weighted harmonic mean of precision and recall. As in [26], we set β = 5 to emphasise recall to penalizing false negative more.
Experimental results. As shown in Tab. 3, the proposed Word2fun II, III and IV outperformed all the baseline methods. Word2Fun with linear parameterization and Word2fun I performed nearly the worst, since they could not learn word-dependent evolution. Word2fun IV achieved worse performance than III, thus showing that adding phase seems unbeneficial to Word2Fun.

heading: 5.2.2 Temporal Analogy
text: The Temporal Analogy task [22] utilizes quadruples (w(1), t(1), w(2), t(2)) to say that “w(1) in year t(1)” is like “w(2) in year t(2)”. To examine the quality of temporal analogy, [26] created a task to investigate equivalences across years. For example, given obama in 2012, we aim to find its equivalent word in 2002. As we know obama was the U.S. president in 2012, its equivalent word
5News articles in NYT dataset are tagged with their ‘sections’ such as ‘Business’, ‘Sports’, ‘World’, and ‘Technology’. For example, we see that amazon occurs 41% of the time in World in 1995, associating strongly with the rainforest, and 50% of the time in Technology in 2012, associating strongly with e-commerce.
in 2002 is bush, who was the U.S. president in 2002. In this way, there are two test sets. The first set (test1) is based on publicly recorded knowledge that lists different names for a particular role, such as U.S. president for each year. Human experts generated the second test (test2) to explore emerging technologies, brands, and significant events (e.g., disease outbreaks and financial crisis), etc. Reciprocal Rank (MRR) and Precision@K (P@K) are used for the evaluation.
Experimental results. As shown in Tab. 4 and Tab. 5, Word2Fun III outperformed all baselines in test2 and some metrics (including P@5, P@10) in test1. In test1, Word2Fun performed worse than Compass-aligned Word2Vec [4] in terms of MRR and P@1; this might be explained by Word2Fun has fewer parameters than [4].

heading: 5.2.3 Semantic Change Detection
text: Semeval-2020 task ‘Unsupervised lexical semantic change detection’ [21] released a dataset for Semantic Change Detection with human-annotated semantically-shift degrees. It provides semanticallyshift degree of thirty seven English words from the timespan 1810-1860 to the timespan 1960-2010. The task aims to predict semantically-shift degree and the performance is evaluated by Pearson and Spearman correlation. We split data of every decade into time bins. We used Word2Fun trained on COHA dataset. For each word, we take the cosine distance between the average word vectors in the first five decades and last five decades as the semantic change degree.
Experimental Results. As shown in Tab. 6, Word2Fun outperformed the five mentioned baselines and the most effective approaches in the Semeval-2020 task. Note that the task participants [19, 17] did not train on the same text collection; they used a subset of COHA and we use a complete one.

heading: 5.3 Analysis of Word2Fun
text: Parameter interpretability To better understand the learned parameters, we will consider Word2Fun III since it was the best performing one. In Word2Fun III, a word wi is represented
as Bi + Ri[sin(Ω (1) i t); cos(Ω (2) i t)] where Bi,Ri,Ωi ∈ RD. There are three cases where the function degrade to a constant function: a) Absolute values of elements in Bi is much larger than that ofRi; b)Ri = 0; or c) Ωi = 0. Intuitively, smallerBi, biggerRi or Ωi indicates that the word wi is sensitive to time. In other words, the meaning of words more likely changes over time in the case of smallerBi, biggerRi or Ωi.
To examine the above intuition, we define the average of the absolute values for these parameters (i.e., Bi,Ri,Ωi ) computed over all the dimensions as indicators for semantically-shift degree. Since in Word2Fun III,Bi is the time-unrelated term whileRi[sin(Ω (1) i t); cos(Ω (2) i t)] is the time related term, we also take the ratio between AVG(|Ri|) and AVG(|Bi|) as an indicator (last row of Tab. 7). Tab. 7 confirms our intuition. Especially, the Pearson correlation values of last two rows are significant with p < 0.05. Although the result in Tab. 7 is not as good as the results in Tab. 6, Tab. 7 is surprising since these indicators do not consider the two specific evaluation timespans and they therefore capture the word evolution “speed” over the whole timespan.
When checking distribution of these parameters, we observe that semantically-shifted words generally have slightly bigger frequencies, bigger amplitudes, and smaller biases. Interestingly, nearly 70% of the sinusoidal functions does not complete a whole period in 20 decades (from 1810s to 2010s).
Visualization of learned functions Learned functions for the top 4 semantically-shifted words and another top 4 semantically-unshifted words6 are visualized in Fig. 1: we selected two dimensions for cosine functions (see Fig. 1a and 1b) and two dimensions for sine functions (see Fig. 1c and 1d) in Word2Fun III. Fig. 1a shows that the function in first dimension learns nearly constant functions for the two semantically-unshifted words (namely, chairman and relationship), indicating the two words will not change their representation over time for this dimension. In Fig. 1b, we could observe that amplitudes for semantic-shifted words are bigger than amplitudes for semantic-unshifted words. In Fig. 1c and 1d with sine functions, the evolution for semantic-shifted words seems to be severe than semantic-unshifted words. In Fig. 1c, it shows some long-term evolution with bigger periods (smaller frequencies). We argue that the proposed Word2Fun has the potential to model long-term word meaning evolution, such long-term evolution might span decades or even centuries [8] especially for general language usage shift.
6We used the annotated test set in Sec. 5.2.3. The top 4 semantically-shifted words are graft, plane, prop, tip; the top 4 semantically-unshifted words are chairman, fiction, relationship, risk.
Case study In Tab. 8, we report some results from the trained word representation in the NYT dataset that ranges from 1990 to 2009. We can see that the word ‘apple’ in the 90s were more about a fruit used to prepare sorbet or chutney. Since the invention of Android in the early 2000s, ‘apple’ has become more related to Android. The same trend can be seen in the case of ‘browser’: it was used in personal computer (e.g., netscape browser) in PC era, and then became more popular in mobile devices (e.g., in Android system) in the mobile Internet era. As for ‘phone’, it typically referred to a traditional telephone in 1990s, and it has become related to 911 (an universal emergency number) since 9/11 attacks. In the mobile internet era, phone is highly related to password since intelligent phones need password while the fixed-line phone does not need. These cases suggest that the proposed Word2Fun can model such semantic change to some extent.

heading: 6 Conclusion and Future Work
text: To smoothly model word meaning evolution in a single process, this paper proposes to model the change of word meaning as explicit functions. With a specific type of function like sinusoidal functions, our approach consists of a sum of many sinusoids to approximate the change of word meaning. The proposed method achieves comparable and sometimes better performance than SOTA dynamic word embedding in time-aware word clustering, temporal analogy, and semantic change detection while with fewer parameters. This work is a preliminary step to develop a sinusoidal function approximation in diachronic word representation. However, we argue this paradigm can inspire more work on a general time-series or sequential modeling research.

heading: Acknowledgments
text: This work has been supported by the Quantum Access and Retrieval Theory (QUARTZ) project, which has received funding from the European Union‘s Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie grant agreement No. 721321.
This work has also in part been supported by MIUR (Italian Ministry of Education, University and Research) through the initiative “Departments of Excellence 2018-2022” (Law 232/2016).
We would like to thank the authors of [4] and [18] for their support in reproducing their experiments. Moreover, we would like to also thank the anonymous reviewers for the very constructive comments.
Societal Impacts
We acknowledge that source data for training has a direct impact on learned embeddings. Documents in source data may involve various biases e.g. (e.g., political, racial, and sexual biases). Therefore, it may also introduce biases from source data to learned embedding that may affect decision making and predictions in downstream tasks. More importantly, such biases may vary over time; manipulating the ratios of documents at different time may have different results in embeddings. We encourage that future studies based on this work should consider this matter.

Review summaries:
This submission addresses the problem of learning diachronic word representations. A new method is proposed that models words as functions and does not rely (it is claimed) on learning representations for discrete time intervals first and then postprocessing them.  The new method is potentially more powerful than prior work since it can in theory approximate any continuous process.  It uses fewer parameters, can model longterm processes and has improved time complexity.  Good performance on a number of tasks is demonstrated. 

This paper proposes a new paradigm for modeling temporal word embeddings. Instead of the commonly prevalent approaches that train static embeddings for documents belonging to each timestamp followed by an alignment step, this work attempts to learn embeddings as a continuous function of time. This work proposes function parametrizations that use sinusoidal polynomials which can model arbitrary continuous functions. The approach is empirically compared against relevant baselines on time-aware clustering, temporal analogy, and semantic change detection.

This paper proposed a new paradigm to learn diachronic word representations, which models words as functions of time, so that the word meaning in a different time could be correlated and evolves within a continuous process. The paper adopted various functions such as linear function and sinusoidal functions to approximate the word meaning evolution in the context of distributed representations. The experimental results show that this approach is promising in the time-aware NLP tasks like time-aware word clustering, temporal analogy, and semantic change detection.

 This paper presents a new approach to modeling word embedding change over time: each word $w_i$ is represented as a function $g_i(t) : N \rightarrow R^D$ that maps a timestamp $t \in \mathbb{N}$ to an embedding $e_{it} \in \mathbb{R}^D$. The function $g_i(t)$ can then be parameterized in a variety of ways; the paper focuses primarily on sinusoidal parameterizations. The paper motivates this approach via the Weierstrass theorem, from which it follows that a real function on the interval [a,b] can be well approximated by a trigonometric polynomial.  The key contribution of this paper is that it breaks out of the typical paradigm that has been used for diacronic word embeddings: i.e., that a separate word embedding $e_{it}$ is explicitly learned for each word $w_i$ and each timestamp $t$. Doing so comes with several drawbacks that this paper addresses: (1) that an alignment between timestamps must be learned (2) that the alignments are (usually) only dependent in a pairwise Markovian fashion, thus missing out on longer time effects (3) that space required is high.  This paper evaluates its method with different parametric functions (e.g. linear, word/time independent, and various sinusoids) on several benchmarks established in prior work for the evaluation of diacrhonic word embeddings and attains (mostly) better results than strong baselines from prior work.