Word2Fun: Modelling Words as Functions for Diachronic Word Representation
Authors: Benyou Wang
Emanuele Di Buccio
Massimo Melucci
Venue: NeurIPS 2021 Poster (2021)
Keywords: Diachronic Word Representation
word vectors
Temporal Analogy
embedding

Abstract: Word meaning may change over time as a reflection of changes in human society. Therefore, modeling time in word representation is necessary for some diachronic tasks. Most existing diachronic word representation approaches train the embeddings separately for each pre-grouped time-stamped corpus and align these embeddings, e.g., by orthogonal projections, vector initialization, temporal referencing, and compass. However, not only does word meaning change in a short time, word meaning may also be subject to evolution over long timespans, thus resulting in a unified continuous process. A recent approach called `DiffTime'  models semantic evolution as functions parameterized by multiple-layer nonlinear neural networks over time. In this paper, we will carry on this line of work by learning explicit functions over time  for each word. Our approach, called `Word2Fun', reduces the space complexity from $\mathcal{O}(TVD)$ to $\mathcal{O}(kVD)$ where $k$  is a small constant ($k \ll T $). In particular, a specific instance based on polynomial functions could provably approximate any function modeling word evolution with a given negligible error thanks to the Weierstrass Approximation Theorem. The effectiveness of the proposed approach is evaluated in diverse tasks including time-aware word clustering, temporal analogy, and semantic change detection. Code at: {\url{https://github.com/wabyking/Word2Fun.git}}. 

Word2Fun: Modelling Words as Functions for Diachronic Word Representation
Benyou Wang', Emanuele Buccio 1.2 and Massimo Melucci'
Department of Information Engineering Department of Statistical Sciences University of Padova, Padova, Italy (wang, dibuccio, melo) @dei. unipd.it
Abstract
Word meaning may change over time as reflection of changes in human so- ciety. Therefore, modeling time in word representation is necessary for some diachronic tasks. Most existing diachronic word representation approaches train the embeddings separately for each pre-grouped time-stamped corpus and align these embeddings, e.g., by orthogonal projections, vector initialization, temporal referencing, and compass. However, not only does word meaning change in a short time, word meaning may also be subject to evolution over long timespans, thus resulting in unified continuous process. recent approach called DDiff- Time' models semantic evolution functions parameterized by multiple-layer nonlinear neural networks over time. In this paper, we will carry on this line work by learning explicit functions over time for each word. Our approach, called 'Word2Fun', reduces the space complexity from DT to ((VD) where k is small constant (k < T). In particular, specific instance based polynomial functions could provably approximate any function modeling word evolution with given negligible
thanks to the Weierstrass Approximation Theorem. The effectiveness the proposed approach is evaluated in diverse tasks including time- aware word clustering, temporal analogy, and semantic change detection. Code https: /ttp::/ggttt.... com hcc.m//aawyynn//oood/uuu.... www.//or....n. git
Introduction
Word meaning changes over time as a reflection of the changes of human society, Not only does word meaning change in a short time, word meaning may also be subject to evolution over long timespans.
The causes of the change of word meaning may be cultural, socictal, or technological (23). Such
lexical semantic change phenomenon has been investigated for some decades 3 23
Modeling the change word meaning recently relies on distributed representations of words 14 and adopting time-specific word vectors as diachronic word representation. The training is based on diachronic text corpora, which are obtained by partitioning the corpora into bins of textual documents labeled by time. A popular training paradigm within such an approach is known as train-and-lle first, the embeddings are trained separately for each pre-grouped sime-stamped corpus; then, the embeddings are aligned by computing, e.g. orthogonal projections 12], vector initialization (I]) temporal referencing [7], parameter regularizers 26 20), aligned compass 4, or latent diffusion
[ The methods above do not model word meaning evolution continuous process, but consider one-hop transformation between pair of timestamps. However, the meaning of some words may smoothly evolve across long-range span, according to mixture of short and long timespans or general according to unknown patterns which cannot straightforwardly be represented only considering one-hop transformations.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

A recent approach called 'DiffTime' [8]] models semantic evolution as functions parameterized by multiple-layer nonlinear neural networks over time. Our paper carries on this line of work by modeling the change of word meaning as implicit functions for diachronic word representation, and considering word meaning evolution as a continuous process. These functions have considerable expressive power since polynomials can approximate any functions with given negligible error thanks to the Weierstrass Approximation Theorem. Even if the proposed approach is general, we will focus on trigonometric functions.
We experimentally show that modelling each word as finite number of trigonometric functions combined together in trigonometric polynomials is an effective approach to modeling the change of word meaning. Our experimental investigation also shows that the proposed method, even using fewer parameters, achieves better performance than the state-of-the-ur (SOTA) diachronic word representation in time-aware word clustering, temporal analogy, and semantic change detection tasks, showing its potential for the research in diachronic word meaning representation and processing.
2 Background
2.1 Problem Definition
Representing words vector space, i.e., a static word embedding 14 h:N-- RR, is a common practice. However, static word embedding cannot model word dynamies over time, which is crucial when considering time-dependent corpora and some tasks, such as those carried out by specialists like linguists sociologists. Diachronic word embedding (DWE) [9] (sometimes called Dynamic Word Embedding [T] assumes that each word has a meaning which may vary with time. One word w; with index i has its meaning represented as a D-dimensional vector Ug,t R a given time t. Therefore, DWE can be formalized D-dimensional vector defined over a real field.
mapping from N N to R., i.e. from the pair (i, t) to
Most methods for DWE separately train embeddings for each time / E N. Note that even if we assume E N, all formalizations can be extended to E RR; practice, the granularity between two consecutive times is fixed, e.g., one year, thus representing & in N is enough. The vectors that
represent V words in time are defined as U. def [U U2.1 Uv.ll E RVD.
2.2 Alignment Issues for Rotation-invarial Embeddings
There are typically two ways to obtain word vectors: one method matrix factorization, the other method is prediction-baseeeeeeeeeeeeeeeeep neural methods; the former method is also related to the latter [13]. When considering the matrix factorization case, a common approach (26 is to approximate a given Positive Pointwise Mutual Information (PPMI) matrix M, IR VVV by means of the V word vectors U,, such that U. example, the entry at row and column j may measure the degree of co-occurrence of the i-th word with the j-word within fixed-size textual windows. The word embeddings can be obtained by either using an eigenvalue method or a matrix factorization method which finds
M,. The entries of M, are measures of relationship between words; for
U arg min VV MilF
However, we have that U',, U. can be an alternative solution of Eq. for any orthogonal rotation matrix R E DxD, Indeed, UU,, = URR (U.,)" = U., (U,,)" since RR. = Therefore, training word embeddings according to Eq. iis non-convex and as a consequence global minimum might not exist.
Moreover, if diachronic word embeddings are trained twice on the same time-specific corpus yet with different random seeds, different solutions may result. also possible that two word vectors corresponding to different times, i.e., U.,, and U U.t, cannot be compared because they might be arbitrarily rotated. One has to align these time-specific word embeddings. The rotation-invariant issue is also present for prediction- word embeddings e.g., Word2Vec [4].

---

Table 1: Comparison of existing diachronic word representation and Word2Fun. refers word index; j refers to dimension index. The target embedding C' E RVD is not considered in this table.
Diachronic word vectors
formalization
Param.
Word2Vec static word embeddings
W
VD
linear projection [
Aligned
vector initialization
Diachronic
discrepancy penalty 20
26
L E RP
TVD
Word2Vec
latent diffusion 1
temporal referencing
aligned by compass
DiffTime [8]
NN (foadd (1), fume (t)), NN
2VD D + +AD
linear function
W Ket
2VD
I (Time2Fun)
B: + sin (Stt)
VD+D
Word2Fun
II (w/ fixed freqs) )
B. + R. (St;; cos((2t)) (Stt)) St 10600
1/4
2VD
III (w/ trainable freqs)
B: + R. (sii((()) ( 2 t); cos(12 ())
3VD
IV (w/ trainable phases)
B. + R. (sin(( (t) t + O ((()))) cos t+
(())]
4VD
2.3 Existing Work of Diachronic Word Embedding
Existing DWE, e.g., 19 considers word meaning evolution in two consecutive time-specific corpora (e.g., 1 and f2), instead of a whole continuous process. They train word vectors for each time-specific
corpus and then align them using orthogonal projections 9 [2]. vector initialization (TI.. temporal
referencing (7], parameter regularizers (26] 20), aligned compass [4], latent diffusion M. Tab. [ summarizes these approaches. The number of parameters of the alignment-based dynamic word
embeddings is proportional to the number T of time bins, resulting in a parameter space O(TV This is notable cost with a long time span, e.g., splitting COHA dataset yearly results in 200 time slices (T = 200). Furthermore, transformations between two any consecutive time bins are independent, namely, U Ui,t, may be totally independent of the one between Ug,t Ug.t even they are close in the time. Note that these approaches might lose some power when modeling such sequential evolution with more than time bins in the event of word meaning that gradually evolves during a longer timespan, c.g., many decades. Lastly, dividing a corpus into separate time bins may lead many smaller training sets. Word vectors in time bin are trained with roughly + training examples instead of all examples. This might be an issue with time-unbalanced dynamic corpora. For example, when there are much fewer documents in a specific year than others, the word-specific word vectors in that year may be distorted because of underfitting; it may therefore negatively affect the long-range transformation of word meaning change.
Dime 18 was the first work to model word meaning evolution as continuous process to the best of our knowledge. It leverages multiple-layer neural networks to aggregate output from word encoder f and a time encoder g. Even if the parameter scale does not increase with a bigger T, DiffTime additionally introduces much more parameters (e.g., D Such learned functions are less interpretable due to the depth and non-linearity of neural networks. We argue that the neural network architecture in [18 is ad hoc without theoretical approximation guarantee for semantic shift.
3
Methodology: Modelling Words as Functions
3.1 Formulation
To smoothly model word meaning change over time we represent each word as continuous function: specific word vector at time t is represented as the values of the function when the variable equals t, which is inspired by [24.] 88 10 25). More formally, our approach aims to learn a mapping f that maps each word w., with index i E N, to functions over time:
f:N (g N - RD)
where y is function over variable E N. Note that the output of y is D-dimensional vector, g(l) E RD. Let us denote f(i) as gr. A word u, at time /. represented as a D-dimensional vector U (() = (t).
3

---

Examples of are linear functions (t) = b+ kt with parameters b, k E RD or sinusoidal functions g(t) = b + sin(wt + 0) with parameters r, w. RR.. Section reports some remarks on the considered instantiations of and how they are linked to previously proposed approaches.
3.2 A Skip-gram Implementation with Compass
The Skip-gram model architecture tries to predict the source context words (surrounding words) given target word (the center word), which was considered to achieve the reverse of what the CBOW model does. Let us denote u as the target word and as the source word, and negatively-sampled target words == {6 It learns a diachronic word representation method f x IR + R and static word mapping h N + RP; (u, t) denotes the diachronic word vector of w. in time
The static word embedding h h(u) introduced to align the diachromic word vectors in different times; it is called a compuss in [4. The compass is used an anchor to project time-specific word embedding different times to unified designated vector space. Our implementation of the compuss differs from [4, since we jointly train the static compass and diachronic word vectors simultaneously, thus avoiding a two-stage training and the compass can be non-fixed the second stage.
The loss function is defined as:
C
Σ Wie
(o)()()(
Σ
I
B.EV
is the sigmoid activation function. The algorithm explained Algo. and the details on the parameterization are reported in Tab.
ALGORITHM 1: Training algorithm for Word2Fun.
Require:
time-stamped corpora C;
diachronic word mapping / (I) N
a static word mapping h N + RD
I: shuffling time-stamped corpora
2: for each epoch do
each document with timestamp 4: for each skip-gram (u, v) in do
do
calculating the loss for the positive sample Lpos log(6(f(u,
6: generating negatively sampled target words V
calculating the loss for negative samples Lass =- log(0-)
8:
back proepprppaggiigg using the loss L = Lpa + Lowg end for
10: end for
11: end for
Which Functions?
4.1 Function Approximation using Polynomials
For skip-gram pair (w;, wy) The similarity degree between a context word w;; and a target word wy in time is calculated as a dot product between the time-specific context embedding f(, and the static target embedding h(j), the latter being the "compass":
(t) f(i.t)A()²
(t) function of time and it measures the similarity between w; and wy over time we argue that semantic meaning evolution be captured by approximating arbitrary between- similarity over time. How to approximate yi. (f) by selecting appropriate parameterization of and h?
If wi and wy appear together f-size window, call (w,, w;) as skip-gram pair in this paper, wi the context word and wy the target word, and vice versa. For PPMI factorization to obtain word vectors, (t) is the changing PPMI between and wy time.

---

Let us recall the Weierstrass Approximation Theorem that states that every continuous real function defined on (a, can be approximated by real polynomial. Note that Theorem also holds for trigonometric polynomial defined in Definition I see Corollary D
Theorem 1. Weierstrass Approximation Theorem. Let F be a continuous real-valued function defined on the real interval (a. b). For any 6 > O, there exists a polynomial such that for all E [a,b], b), we have [F(z) - ~()) <.
Definition 1. A trigonometric polynomial of degree D is an expression of the form A Ek=1 Og cos(kr) + 3x sin(kx), where A, a1,...;;D.. .i...., BD E R.
D
Corollary 1. Approximation by trigonometric polynomials For every continuous 2n 2 function F R - R. defined on the real interval [0, 2 and for any >> 0, there exists a trigonometric polynomial P such that for all E [0, 2 we have FF)) P(x) <.
Corollary 2. The trigonometric polynomials span
cos cOS ... .iin( (a), cos(Dz)) is dense a-b cos 2=Dz is dense in Cla, b).
in ClO, 2T] iff span { sin 2
2ns sin 2xDr
a-b
know that {0, sin 008 008., sin Dx), cos [Dr}} spans subspace that can approximate any periodi- cal continuous functions defined on (0, 2 i.e. it is dense in ClO, 2-.. By Corollary 2 one can conclude that weighted sum of trigonometric functions with appropriate periods can approximate any periodical continuous functions defined in an arbitrary closed interval [16].
From Corollary
trigonometric polynomial
4.2 Sinusoidal Parameterization in Word2Fun
Since the static embedding h(j) is not related to time we consider f(i, () with sinusoidal paramer- ization (a mixture of cosine and sine functions plus a bias term). Then Eq. will result in
B + R sin(S
C₁a2
B + R.22 (Os
C3,2
(t,(t) = -(;)))/())
B + R. sin n
CjP--I C2
B.D + R. cos o
(4)
D
B...C,K
R122k- Cj,2k sin + R.22 C,22 cos (Qxt)
1=1
1=1
Therefore, (t) is weighted sum of sinusoidal functions plus constant term Σb B By replacing the coefficients with as.j.k and Big,k- we can rewrite Eq. as (i, A
Ex Oi.j. sSn(t) + Bi,s cos(Slut); {oı,j,k and (B.,, km are the coefficients and (Re)kz are the corresponding frequencies. Following the argument in [5] when discussing approxi- mation properties of sine and cosine activation functions, since linear combinations of sine and cosine generate finite trigonometric polynomials that have approximation properties described in Section 4.1, the dot product between the diachronic embedding fii, t) and the static target embedding h(i), denoted as yi.r(), can capture any evolving relations between arbitrary skip-gram pairs.
2
By iteratively training all skip-gram pairs, Word2Fun can model complicated semantic change across time. Intuitively, small frequencies would reflect some long-range evolution, while some big frequencies would capture short-range evolution. Such periodical property would allow such functions to capture long enough evolution without considering boundedness issues.
4.3 Investigated Function Parameterizations
Linear functions. real polynomial naturally induce: U.,: = (t) b, + k, PIT.
Since ? computationally expensive and sometimes meets the overflow issue, we choose simpler case with a linear function parameterization of Word2fun: U gi(() = b. k,d.
g is the element-wine wise multiplication
5

---

Table 2: Statistics of diachronic corpora.
corpus
num. of tokens
time range
time granularity
T
Vocab
COHA 6
472M
1810 2009
every decade
20
43, 734
New York Times
26
105M
1990 2016
yearly
27
20, 314
Sinusoidal functions. In this work, we investigate the sinusoidal parameterization. We use a combination of sine and cosine functions; i.e., half of dimensions are parameterized by sine functions and half of them are parameterized by cosine functions. Considering the domain of time & E N and its range in this paper, we adopt set of frequencies {w,} (24] that is different from standard trigonometric polynomial. The Word2Fun II is defined as below:
3/4
JE 1
(t) b, + r:
S2 10000
(5)
Note we add a static component b, that is independent from time. In Word2Fun III we also learn word-specific frequencies: U (t) b, + rs(sin(w)
t); cos())) (o.()) Word2Fun IV adds
trainable word-specific phases: U.f = 9: (t) = b, + rs (sin((w) ( + o(t)); cos (w (2) 0
((())) see
(1) Tab. w E ER is different from w
(2) R _ of E E is different from 0, R
Standard static word vectors, c.g., [1]. can be considered as constant functions, a special case of Word2fun since b; is time-agnostic word vector. For instance, (1) in the case of the linear function, k; = 0: (2) in the case of sinusoidal function, r; = 0 or wi is small enough. Both these cases results in g. b;- The additional parameters Wi and r; expected capture the word meaning evolution.
Fun: purely encoding time as functions. time mapping:
We can separately learn a word mapping and a
fword N - RD, fume N - RD
(6)
Then w, at time t can be represented as U. = (i) + fume (t), where füme can be function over time, e.g., frme Ktt frie = sin
the latter corresponds Word2Fun I in Tab. [ Time2Fun is close to 18 which represents time as continuous variable and model a word's usage as a function of time. also explores to use time as an extra feature for various tasks.
Experiments
5.1 Data and Experimental Settings
The diachronic corpora used in this paper are reported in Tab. 2 The Corpus of Historical American English (COHA) (6 is the largest structured corpus of historical English (the 18 1810s-2010s, contains more than 475 million words and balanced by genre decade by decade. The New York Times (NYT) (26) contains 99,872 articles published between January 1990 and July 2016; besides the article text, metadata including title, author, release date, and section label were also collected.
Corpora.
Baselines. We choose the baselines from (26. Static Word2Vec: the standard word2vec embeddings [14], trained on the entire corpus and ignoring time information. Transformed Word2Vec [12]: the embeddings are first trained separately by factorizing PPMI matrix for each year, and then transformed by optimizing a linear transformation matrix which minimizes the distance between two consequent time-stamped trained embeddings for the k nearest words' embeddings to the querying words. Aligned Word2Vec [9] the embeddings are first trained by factorizing the PPMI matrix for each year t, and then aligned by searching for the best orthonormal transformation between two consequent time-stamped trained embeddings. Dynamic Word Embedding (26] learns word embedding by factorizing PPMI matrix in individual time; plus, it imposes regularizer to encourage two subsequent word embedding being similar for alignment. Besides those baselines, we considered DiffTime [8 (described in Sec. time-specific context embeddings. The former has embedding size of 1550 and its hidden dimension is 50, and the latter has embedding size of 100; the both have more parameters than Word2Fun.
Sec. (.3) and Compass Aligned Word Embedding [4] that exploits fixed target embeddings to align
"Sinusoidal functions introduce infinite local minima due to the periodicity of sinusoidal functions [5] but empirically found it performs well [10] and leave the above issue future work.

---

Table Experimental results of time-aware word clustering.
10 Clusters
15 Clusters
20 Clusters
Method
NMI
Fg
NMI
F2
NMI
Fg
Global/static word vector [4
0.6736
0.6163
0.6867
0.7147
0.6713
0.7214
Transformed Word2Vec
[2
0.5175
0,4584
0.5221
0.5072
0.5130
0.5373
Aligned Word2Vee [9]
0.6580
0.6530
0.6618
0.7115
0.6386
0.7187
Dynamic Word2 Vee 26
0.7175
0.6949
0.7162
0.7515
0.6906
0.7585
Compass ali aligned Word2 Vec 4
0,5191
0.3750
0.5062
0,4051
0,5077
0.4331
DiffTime [8]
0.6945
0.6723
0.6727
0.7161
0,6576
0.7275
Word2Fun linear
0.1676
0.1813
0.2826
0,3035
0.2473
0.2932
Word2Fun I (Time2Fun)
0.1703
0.1783
0.2691
0.2680
0.2842
0.2649
Word2Fun II
0.7281
0.7147
0.7181
0.7645
0.7012
0.7616
Word2Fun III
0.7233
0.7080
0.7086
0.7701
0.6980
0.7630
Word2Fun IV
0.7111
0.6913
0.7023
0.7451
0.6823
0.7602
Experimental settings. We removed words that appear less than 200 times, as (26] did. We used the Adam optimizer with a learning rate of 0.0025. The batch size can be as big as it achieves the upper bound of memory of GPU. The model converged in less than 20 epochs with drop in terms of loss. We computed the average performance on the last three saved checkpoints. For sinusoidal functions, we used mixture of cosine functions and sine functions. The dimension was 100; we preliminary found that models with bigger dimensions (e.g. 200 or 300) can slightly improve performance. Word2Fun with all settings additional computation for obtaining word vectors. However, all methods include Word2 Vec and Word2Fun took 10-15 minutes for one epoch in single Nvidia V100 32G GPU for NYT dataset.
slightly slower than the standard Word2 Vec due to
5.2
Quantitative Evaluation
5.2.1
Time-aware Word Clustering
The time-aware word clustering task was proposed in (26] and exploits the NYT dataset. If one word was extremely frequent in a particular section, it was labeled as most-used in that section 55 Such section annotation was used evaluate the clustering results. Across 11 sections, there were 1,888 triplets denoted as (w; ti, s;) to indicate that word w, in year t; was associated with section sç.
We apply spherical K-means on learned time- imee-specific word vectors using cosine distance, with K = 10, 15, and 20 clusters. As in [26], we used Normalized Mutual Information (NMI) and Fg- measure to measure the consistency between section label and clustering results. NMI defined as NMI(C, S) = 2MI(C, ()) + E(S)), where S denotes section labels (fs;)) for word-year pairs ([w;, 44) and C denotes clustering categories (fc:)) for these word-year pairs [[w,, 4)] MI(:;)) and B(-) are the functions to compute Mutual Information and Entropy respectively. F3= 34P+R measures the effectiveness as 3-weighted harmonic mean of precision and recall. As in [26]. we set to emphasise recall to penalizing false negative more.
1RP
Experimental results. As shown in Tab. 3 the proposed Word2fun II, III and IV outperformed all the baseline methods. Word2Fun with linear parameterization and Word2fun I performed nearly the worst, since they could not learn word-dependent evolution. Word2fun IV achieved worse performance than III, thus showing that adding phase seems unbeneficial to Word2Fun.
5.2.2 Temporal Analogy
The Temporal Analogy task 22 utilizes quadruples (w()) {(1), (2) x(2)) to say that "w(() in year
{(1)" is like (2) in year (2) To examine the quality of temporal analogy, [26) created task to investigate equivalences across years. For example, given obama in 2012, we aim to find its equivalent word in 2002. As we know obama was the U.S. president in 2012, its equivalent word
NNws articles in NYT dataset are tagged with their 'sections' such as 'Business', 'Sports', 'World', and "Technology". For example, we see that amazon occurs 41% % of the time in World in 1995, associating strongly with the rainforest, and 50% of the time in Technology in 2012, associating strongly with e-commerce.
7

---

Table 4: Experimental results of temporal analogy testl
Method
MRR
P@ I
P@3
P@5
P@ 10
Global/static Word2 Vec
14
0.3560
0.2664
0,4210
0,4774
0.5612
Transformed Word2Vee
122
0.0920
0.0500
0.1168
0.1482
0.1910
Aligned Verd2Vec g
0.1582
0.1066
0.1814
0.2241
0.2953
Dynamic Word2Vec 226
0.4222
0.3306
0.4854
0.5488
0.6191
Compass aligned Word2 Vec
0.481
0.404
0.534
0.582
0.636
DiffTime [8]
0.3759
0.2738
0,4200
0.5070
0.5912
Word2Fun linear
0.3016
0.2649
0.3255
0.3426
0.3630
Word2Fun (Time2Fun)
0.3735
0.2646
0.4300
0.4955
0.5874
Word2Fun
0,4061
0.2756
0,4916
0,5614
0.6434
Word2Fun III
0.4354
0.3076
0.5330
0.5837
0.6647
Word2Fun IV
0.4208
0.2954
0.5076
0.5715
0.6470
Table 5: Experimental results of temporal analogy text2
Method
MRR
P@
P@3
P@5
P@ 10
Global/static Word2
14
0,0472
0.0000
0.0787
0.0787
0.2022
Transformed Word2
11
0,0664
0.0404
0.0764
0.0989
0.1438
Aligned Word2Vec
0.0500
0.0225
0,0517
0.0787
0.1416
Dynamic Werd2
0. 0.1444
0.0764
0.1596
0.2202
0.3820
Compass Aligned Word Embedding 44
0.1361
0.0749
0.1918
0.2904
0.3918
DiffTime [18]
0.1259
0.0055
0.1562
0.2219
0.3973
Word2Fun linear
0.0425
0.0137
0.0384
0.0630
0.1014
Word2Fun (Time2Fun)
0,0992
0.0000
0.1315
0.1726
0,2849
Word2Fun II
0.1194
0.0358
0.1075
0.2219
0.3863
Word2Fun III
0.1824
0.0795
0.1973
0.2932
0,4164
Word2Fun IV
0.1536
0.0548
0.1562
0.2411
0.3918
in 2002 is bush, who was the U.S. president in 2002. In this way, there are two test sets. The first set (Iestl) is based on publicly recorded knowledge that lists different names for a particular role, such as U.S. president for each year. Human experts generated the second test (test2) to explore emerging technologies, brands, and significant events (c.g., discase outbreaks and financial crisis), etc. Reciprocal Rank (MRR) and Precision K (P@K) K) are used for the evaluation.
Experimental results. As shown in Tab. and Tab. Word2Fun III outperformed all baselines in test2 and some metrics (including P@5, P@@ 10) in test/. In testl, Word2Fun performed worse than Compass-aligned Word2 4 in terms of MRR and P@ this might be explained by Word2Fun has fewer parameters than
5.2.3 Semantic Change Detection
2202 task 'Unsupervised lexical semantic change detection' 21 released a dataset for Se- mantic Change Detection with human-annotate semantically-shift degrees. It provides semantically- shift degree of thirty seven English words from the timespan 1810- 1860 to the timespan 1960-2010. The task aims to predict semantically-shin degree and the performance is evaluated by Pearson and Spearman correlation. We split data of every decade into time bins. We used Word2Fun trained on COHA dataset. For each word, we take the cosine distance between the average word vectors in the first five decades and last five decades as the semantic change degree.
Experimental Results. As shown in Tab. 6 Word2Fun outperformed the five mentioned baselines and the most effective approaches in the Semeval-2020 task. Note that the task participants [19 17 did not train the same text collection; they used a subset of COHA and we use complete one.
5.3 Analysis of Word2Fun
Parameter interpretability To better understand the learned parameters, we will consider Word2Fun III since it was the best performing one. In Word2Fun III, a word w; is represented

---

Table Semantic change detection. Baselines in the first group are implemented by this work.
models
Pearson
Spearman
Global/static Word2Vec
14
nan
nan
Transformed Ver
12
0.0727
0.0865
Aligned Word2Vec g
0.3333
0.3083
Dynamic Werd2 [26]
0.2727
0.2877
Compass aligned word embedding 4
0.3199
0.2567
Diff Time [18]
0.3504
0.4146
Word2Fun linear
-0.1200
-0.0790
Word2Fun (Time2Fun)
0.3925
0.4550
Word2Fun II
0,4478
0,5038
Word2Fun III
0.5355
0,4057
Word2Fun IV
0.4483
0.3578
multilingual BERT [9] (SemEval-22022 Ist)
0.436
ensemble between aligned Word2 Vec and BERT 17 (SemEEal-22200 2nd)
0.422
Table 7: Learned parameters for semantic change detection. AVG means average for a vector.
Learned parameters
Definition
Pearson
Spearman
frequencies term
(V
0.0958
0.1748
amplitude term bias term
AVG R.
0.1213
0.0474
AVG
-0.3640
-0,2636
ratio between the amplitude and bias terms
AVG R. /..)
0.4141
0,2358
as B, + R. R, (S2 (1) function degrade to constant function: a) Absolute values of elements in B, is much larger than that of R2; b) R, = 0: or e) S2 = 0. Intuitively, smaller B., bigger R. or S2 indicates that the word w; is sensitive time. In other words, the meaning of words more likely changes over time in the case of smaller B, bigger R² or Q4.
t); cos( t)] where B, R, Qr E RD. There are three cases where the
To examine the above intuition, we define the average of the absolute values for these parameters (i.e., B, R2 S2; ) computed over all the dimensions as indicators for semantically-shii degree. Since in Word2Fun III, B, is the time-unrelated term while R, [sin related term, we also take the ratio between (V)() and AVG( [B)) as an indicator (last row of Tab. 7). Tab. conffrms our intuition. Especially, the Pearson correlation values of last two rows are sigmificant with p < 0.05. Although the result in Tab. iss not as good as the results in Tab, 6 Tab. is surprising since these indicators do not consider the two specific evaluation timespans and they therefore capture the word evolution "speed" over the whole timespan.
(1) t)) cos t)] is the time
When checking distribution of these parameters, we observe that semantically-shifted words generally have slightly bigger frequencies, bigger amplitudes, and smaller biases. Interestingly, nearly 70% of the sinusoidal functions does not complete a whole period in 20 decades (from 18 10s to 2010s).
Visualization of learned functions another top mmnantically-unshifeo words are visualized in Fig. | we selected two dimensions for cosine functions (see Fig. aadand [b) and two dimensions for sine functions (see Fig. I ld) in Word2Fun III. Fig. Ta shows that the function in first dimension learns nearly constant functions for the two semantically lyntiicffl--sssietee words (namely, cha i rman and relat iataiionn Lonshi p). indicating the two words will not change their representation over time for this dimension. In Fig. [b] we could observe that amplitudes for semantic-shifted words are bigger than amplitudes for semantic-unshifted words. In Fig. [ ld with sine functions, the evolution for semantic-shifted words seems to be severe than semantic- semantic-usshfted words. In Fig. Ic it shows some long- -eem evolution with bigger periods (smaller frequencies). We argue that the proposed Word2Fun has the potential to model long-term word meaning evolution, such long-term evolution might span decades or even centuries g especially for general language usage shift.
Learned functions for the top 4 semantically
words and
'We used the annotated test set in Sec. 5.2.3 prop, tip; the top 4 semantically unshiifed words are chairman, fiction, relationshie risk.
The top 4 semantically shifted words are graft, plane,

---

1810s
1990s
1890:
(a) The 1-st dimension with cosine functions.
The 13-th dimension with cosine functions.
reletossn1p
changran
1810s
1060s
1.9925
1610s
1860:
1992s
2010s
The 35-th dimension with sine functions.
(d) The 11-th dimension with sine functions.
Figure 1: We show some learned functions for top 4 semantically- shifted words in purple and top cemantically-unshifee words blue, for Word2Fun III with dimension. The first half dimensions (1-25) use cosine functions while the last half dimensions (26-50) use sine functions.
Table 8: Most similar words for apple, european, phone, and browser from the year 1990 (denoted as 90) to 2016 (denoted 16). All words are lower cased.
word
14
apple
sorbet
chuiney
android
Detscape
phone
telephone
911
passwund
curogean
republics
hollunde
republics
Case study In Tab. 8 report some results from the trained word representation in the NYT dataset that ranges from 1990 to 2009. We can see that the word 'apple' in the 90s were more about fruit used to prepare sorber chutney. Since the invention of Android in the early 2000s, 'apple' has become more related to Android. The same trend can be seen in the case of 'browser': was used in personal computer (e.g., netscape browser) in PC era, and then became more popular in mobile devices (e.g., Android system) in the mobile Internet era. As for 'phone', it typically referred traditional telephone in 1990s, and it has become related to 9/ (an universal emergency number) since 9/11 attacks. In the mobile internet era, phone is highly related to password since intelligent phones need password while the fixed-line phone does need. These cases suggest that the proposed Word2Fun can model such semantic change to some extent.
6 Conclusion and Future Work
To smoothly model word meaning evolution in single process, this paper proposes to model the change of word meaning explicit functions. With specific type of function like sinusoidal functions, our approach consists of sum of many sinusoids to approximate the change word meaning. The proposed method achieves comparable and sometimes better performance than SOTA dynamic word embedding in time-aware word clustering, temporal analogy, and semantic change detection while with fewer parameters. This work is a preliminary step to develop sinusoidal function approximation diachronic word representation. However, we argue this paradigm can inspire more work general time-series sequential modeling research.
10

---

Acknowledgments
This work has been supported by the Quantum Access and Retrieval Theory (QUARTZ) project, which has received funding from the European Union's s Horizon 2020 research and innovation programme under the Marie Sklodowska-Curie grant agreement No. 721321.
This work has also in part been supported by MIUR (Italian Ministry of Education, University and Research) through the initiative "Departments of Excellence 2018-2022" (Law 232/2016).
We would like to thank the authors of [4] and 18] for their support in reproducing their experiments. Moreover, we would like also thank the anonymous reviewers for the very constructive comments.
Societal Impacts
We acknowledge that source data for training has a direct impact learned embeddings. Documents in source data may involve various biases c.g. (c.g., political, racial, and sexual biases). Therefore, may also introduce biases from source data to learned embedding that may affect decision making and predictions in downstream tasks. More importantly, such biases may vary over time; manipulating the ratios of documents at different time may have different results in embeddings. We encourage that future studies based on this work should consider this matter.
References
[1] Robert Bamler and Stephan Mandt. Dynamic word embeddings. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 380-389, PMLR, 06-11 Aug 2017.
[2] Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Janvin. A neural probabilistic language model. The Journal of Machine Learning Research, 3:1137-1155, 2003.
[3] Ronnie Cann, Ruth Kempson, and Eleni Gregoromichelaki. Semantics: An introduction to meaning in language. Cambridge University Press, 2009.
[4] Valerio Di Carlo, Federico Bianchi, and Matteo Palmonari. Training temporal word embeddings with compass. In Proceedings of The Thirty-Thirl AAAI Conference on Artificial Intelligence, AAAI, pages 6326-6334. AAAI Press, 2019.
[5] George Cybenko. Approximation by superpositions sigmoidal function. Mathematics of control, signals and systems, 2(4):303-314 1989.
[6] Mark Davies. Expanding horizons in historical linguistics with the 400-million word corpus of historical american english. Corpora, 7(2):121-157, 2012.
[7] Haim Dubossarsky, Simon Hengchen, Nina Tahmasebi, and Dominik Schlechtweg. Time-out: Temporal referencing for robust modeling of lexical semantic change. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 457-470 Florence, Italy, July 2019. Association for Computational Linguistics.
[8] Jacob Eisenstein. Measuring and modeling language change. In Proceedings the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Tutorials, pages 14, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.
[9] William L. Hamilton, Jure Leskovec, and Dan Jurafsky. Diachronic word embeddings reveal statistical laws of semantic change. In Proceedings of the 54th Anmual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), pages 1489-1501, Berlin, Germany, August 2016. Association for Computational Linguistics.
[10] Seyed Mehran Kazemi, Rishab Goel, Sepehr Eghbali, Janahan Ramanan, Jaspreet Sahota, Sanjay Thakur, Stella Wu, Cathal Smyth, Pascal Poupart, and Marcus Brubaker. Time2vec: Learning vector representation of time. arXiv preprint arXiv:1907.055: 2019.
11

---

[11] Yoon Kim, Yi-I Chiu, Kentaro Hanaki, Darshan Hegde, and Slav Petrov. Temporal analysis of language through neural language models. In Proceedings of the ACL 2014 Workshop on Language Technologies and Computational Social Science, pages 61-65, Baltimore, MD, USA, June 2014. Association for Computational Linguisties.
[12] Vivek Kulkarni, Rami AI-Rfou, Bryan Perozzi, and Steven Skiena. Statistically significant detection of linguistic change. In Proceedings of the 24th International Conference on World Wide Web, pages 625-635,22 2015.
[13]
Omer Levy and Yoav Goldberg. Neural word embedding as implicit matrix factorization. In Proceedings of the 27th International Conference Neural Information Processing Systems Volume 2, NIPS' 14, page 2177-2185, Cambridge, MA, USA, 2014. MIT Press.
[14] Tomás Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. In Yoshua Bengio and Yann LeCun, editors, 1st International Conference on Learning Representations, ICLR 2013, Scottsdale, Arizona, USA, May 2-4, 2013, Workshop Track Proceedings, 2013.
[15] Giambatista Parascandolo, Heikki Huttunen, and Tuomas Virtanen. Taming the waves: Sine as activation function in deep neural networks. https//pppaavveee..ee/rffflsssss visited on October 2021, 2016.
[16] Allan Pinkus. Weierstrass and approximation theory. Journal of Approximation Theory, 107(1):1-66, 2000.
[17] Martin Pömsl and Roman Lyapin. Circe at SemEval-2020 Task 1: Ensembling context-free and context-dependent word representations. ACL, 2020.
[18] Alex Rosenfeld and Katrin Erk. Deep neural models of semantic shift. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume I (Long Papers), pages 474 484, 2018.
[19] David Rother, Thomas Haider, and Steffen Eger. Cmce at SemEval-2020 Task 1: Clustering on manifolds of contextualized embeddings detect historical meaning shifts. In Proceedings of the Fourteenth Workshop on Semantic Evaluation, pages 187-193, 2020.
[20] Maja Rudolph and David Blei. Dynamic embeddings for language evolution. In Proceedings of the 2018 World Wide Web Conference, pages 1003-1011, 2018
[21] Dominik Schlechtweg, Barbara McGillivray, Simon Hengchen, Haim Dubossarsky, and Nina Tahmasebi. SemEval-2020 Task 1: Unsupervised lexical semantic change detection. SemEval, 2020.
[22] Terrence Szymanski. Temporal word analogies: Identifying lexical replacement with diachronic word embeddings. In Proceedings of the 55th annual meeting of the association for computa- tional linguistics (volume 2: Short papers), pages 448-453, 2017.
[23] Nina Tahmasebi, Lars Borin, and Adam Jatowt. Survey of computational approaches to lexical semantic change, 2019.
[24] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. NIPS 2017, 2017.
[25] Benyou Wang, Lifeng Shang, Christina Lioma, Xin Jiang, Hao Yang, Qun Liu, and Jakob Grue Simonsen. On position embeddings in BERT. In International Conference on Learning Representations, 2021.
[26] Zijun Yao, Yifan Sun, Weicong Ding, Nikhil Rao, and Hui Xiong. Dynamic word embeddings for evolving semantic discovery. In Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining, WSDM '18 page 673-681, New York, NY, USA, 201 8. Association for Computing Machinery.
[27] Matt Young. The Stone-Weierstre theorem. In MATH 328 Notes. Queen's University at Kingston, 2006.
12


Reviews:
[{"review_id": "ck5JWJ5gM9e", "reviewer": "Reviewer_dgc4", "summary": "This paper proposed a new paradigm to learn diachronic word representations, which models words as functions of time, so that the word meaning in a different time could be correlated and evolves within a continuous process. The paper adopted various functions such as linear function and sinusoidal functions to approximate the word meaning evolution in the context of distributed representations. The experimental results show that this approach is promising in the time-aware NLP tasks like time-aware word clustering, temporal analogy, and semantic change detection.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Generally, the topics of the paper and the idea of modeling words as functions of time are super interesting. The paper is easy to follow until section 4.1, before which all the motivations are clear and straightforward. My main concern is why not use a neural network to learn the approximation in Eq. 3? E.g., adding a time embedding to a distributed model. I suggest that a more convincing motivation behind the Polynominal approximation should be provided.\n\nStrengths:\n* The paper provided a really good background section, which helps the reader better understand the problem settings and related work.\n* The experiments includes lots of time-aware tasks that validates the benefits of the methods.\n\nWeaknesses\n* Word2Fun does not improve significantly compared to Compass aligned Word2Vec in Table 4 and Table 5.\n* The motivation of using polynomials is not clear.\n\nQuestions:\n* Did you consider polysemous words at the same timestamp?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Please see the main review for further details.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "Ls7e3SeXBE", "reviewer": "Reviewer_6eKb", "summary": "This submission addresses the problem of learning diachronic\nword representations. A new method is proposed that models\nwords as functions and does not rely (it is claimed) on\nlearning representations for discrete time intervals first\nand then postprocessing them.  The new method is potentially\nmore powerful than prior work since it can in theory\napproximate any continuous process.  It uses fewer\nparameters, can model longterm processes and has improved\ntime complexity.  Good performance on a number of tasks is\ndemonstrated.\n", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Current methods for learning diachronic representations all\nhave some drawbacks. So it is great to see this new\nproposal, which is in principle more powerful than previous\nwork (since it can model any continuous function) and seems\nto be more sample-efficient (by not partitioning the\noverall learning problem into separate small learning\nproblems first).\n\nHowever, I have several concerns about the details of the\npaper, which are discussed below. Unless these concerns are\naddressed, I would lean against accepting the paper.\n\nDISCRETE VS CONTINUOUS MODELS\n\nI'm not sure I understood the claim that all prior work is\nlimited to comparing consecutive time slices through a\nMarkov process. Vector space alignment approaches usually\nalign all time slices to a single designated vector\nspace. This does not impose a Markovian structure on the\nrepresentations. \n\nOn lines 59 to 62, an argument is made that it doesn't\nmatter whether you model the process as discrete or\ncontinuous. But Equation (2) is discrete in time. Surely,\nthe claim is not that how you discretize time for Equation\n(2) does not matter? Could you say more about what your\ndiscretization is? (e.g., how many time slices you use)\n\nPRIOR WORK\n\nThere is at least one other approach that is not based on a\ntime-stamped corpus:\n\nhttps://aclanthology.org/N18-1044/\n\nThis paper should be discussed and compared against.\n\nTIME COMPLEXITY\n\nThe argument that O(TVD) gives you better complexity that\nO(kVD) seems weak to me. T is usually relatively\nsmall. Maybe this could be backed up by giving examples\nwhere the number of time slices is large?\n\nMODELING OF LONGTERM PROCESSES\n\n\"do not consider meaning evolution in longer timespan\"\n(lines 6/7)\n\nIt would be an interesting claim to say that there are such\nlonger timespan meaning evolutions that the proposed method\ncan model whereas prior work cannot.\n\nHowever, I find this unconvincing without any analysis. It\nseems to me that all prior work can model simple longer\ntimespan meaning evolutions, e.g., if they are simple linear\nevolutions. So the question is not so much: can you model\nlonger timespan evolutions, but rather: do *complex* longer\ntimespan evolutions occur in reality? The paper lacks\nanalysis relevant to this question.\n\nMODELING \"POWER\" OF PROPOSED METHOD\n\nThe proposed method can in principle provide a more\npowerful modeling framework than prior work. However, there\nis no analysis showing that it does anything interesting\nover and above prior work in my opinion.\n\nThe authors do not seem to be very knowledgeable in the\nlinguistic details of diachronic change. A good reference to\nrefer to would be this:\n\nPeter Koch. 2016. Meaning change and semantic shifts. In\nPäivi Juvonen and Maria\nKoptjevskaja-Tamm, editors, The lexical typology of semantic\nshifts, pages 21–66. De Gruyter,\nBerlin.\n\nKoch proposes that diachronic change consists of a process\nwhere a word with the initial meaning M1 acquires a second\nmeaning M2. M2 initially may have frequency close to 0, but\nthen increases over time. So one theory of diachronic change\nis as a process of a change in frequency distribution of\nunderlying senses.\n\nOne could derive many interesting hypotheses from this\ntheory and test them using word2fun.\n\nFor example, are there any *individual* words that mix\nshortterm and longterm evolutionary components in their\nmeaning change? If it's a simple frequency distribution\nchange, then maybe not.\n\nDoes an individual word show different speeds in meaning\nchange over time? There should be examples of that, e.g., a\nvery slow increase of M2 initially and then a quick fading\nof M1 later.\n\nDo different words have different time scales with respect\nto meaning change? I would suspect yes, and the model (if it\nactually lives up to its promise and works as intended)\nshould be able to show this.\n\nWithout any analysis of what the model is actually learning\n(beyond some performance numbers), it is not possible for me\nto determine whether it is doing something interesting\nbeyond prior work or not.\n\nEVALUATION\n\nMy understanding of the semantic change evaluation is that a\nnon-standard setup is chosen, so it is difficult to compare\nmodel performance to prior work. I would find it necessary\nthat experiments are run that guarantee direct comparability\nto Semeval-2020.\n\nMINOR POINTS\n\n\"Because of the equivalence between prediction-based word\nembedding and matrix factorization[12]\"\n\nIt is generally believed that the proof in [12] is not\ncorrect, so the equivalence should not be presented as\ntruth, but as a conjecture.\n\nThe notion of compass (line 129) is not sufficiently\nintroduced and explained.\n\n\n\n\n\n\n\nRESPONSE TO AUTHOR RESPONSE AND REVIEWER DISCUSSION\n\n\nI would like to thank the reviewers for responding to my\nreview!\n\nI'm not very familiar with the SemEval dataset, but what you\nwrite makes sense. It is still unfortunate that there is no\ndirect comparison to prior published numbers.\n\n> We believe that the revised paper has been improved,\n> unfortunately, the submit system does not allow us to\n> update the paper (as ICLR did).\n\nThis is indeed a big issue. I could imagine that I would\nview the paper more favorably if I could read the updated\nversion, but since that's not possible, I don't feel\nconfident about upgrading my review.\n\nIt's a particularly big issue for this:\n\n> Analysis of what the model is actually learning We will add\n> a new section and Appendix B to quantitatively show and\n> visualize what the model is actually learning.\n\nSince I cannot look at this, I don't feel comfortable\nallowing it to influence my recommendation.\n\n> The polysemous (multiple meanings for a single word) issue\n> cannot easily be solved by typical word vectors since they\n> adopt a paradigm of \"one vector per word\", one may refer to\n> \"contextualized word embedding\" like ELMO, BERT for this\n> issue.\n\nThis is a longer discussion, but it is actually possible to\nanalyze single-vectors for polysemy. For example, if sense A\ndominates in the first period and sense B dominates in the\nlast period, then the first single-vector will be close to\nwords with meanings similar to sense A and the last\nsingle-vector will be close to words with meanings similar\nto sense B.\n\nI don't feel my concerns about Rosenfeld & ErK and about\nanalysis were addressed. I concur with my fellow reviewers\non these points:\n\noVhR: I would worry that if this paper were accepted they\nmight continue to downplay prior work in the same way,\nrather than appropriately attributing the main idea of\nmodeling words as functions to R&E'18 as deserved.\n\nrTwC: While, the analysis that finds that most of the words\ndo not complete a full period is interesting, I agree with\nother reviewers that such analysis regarding several aspects\nof the diachronic change is crucial to better understanding\nof the effectiveness of the approach. In fact, in response\nto R6eKb, the authors' proposed metric of correlation b/w\ntemporal change via analyzing gradients (not in the paper)\nexhibits low correlation values. While this doesn't\nnecessarily mean that the proposed approach doesn't work,\nthis certainly underscores the importance of a more thorough\nanalysis to understand the behavior of the proposed\napproach.\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"ethical_concerns": "None.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "FTANny4P54", "reviewer": "Reviewer_rTwC", "summary": "This paper proposes a new paradigm for modeling temporal word embeddings. Instead of the commonly prevalent approaches that train static embeddings for documents belonging to each timestamp followed by an alignment step, this work attempts to learn embeddings as a continuous function of time. This work proposes function parametrizations that use sinusoidal polynomials which can model arbitrary continuous functions. The approach is empirically compared against relevant baselines on time-aware clustering, temporal analogy, and semantic change detection.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Positives:\n\n-- The proposed approach convincingly outperforms most of the baselines while being competitive with dynamic word embeddings. The various ablation of the proposed approach also highlight that modeling frequency parameters are more useful than explicitly modeling phase parameters.\n\n-- The proposed approach is a natural way of thinking about temporal embeddings and is a logical succession to the traditional temporal word embeddings.\n\nNegatives:\n\n-- The loss function seems to work but it seems heuristics-oriented and the motivation behind the loss is weak. What is the motivation behind using both time-sensitive f(.) and compass embeddings h(.) in the loss? Does it relate to any other probabilistic objective functions?\n\n-- In table 4, the proposed approaches trail by compass aligned word2vec by a significant margin on test1. While the prposed approach outperforms the same baselines appreciably on test 2 in table 5. Hence, the proposed explantation of compass embeddings using word2vec implementation is unsatisfactory. More ablative and qualitative analysis focusing on differences between test2 and test1 would shed more light on the differences in behavior of the proposed approach and compass aligned embeddings.\n\n-- Analysis of learned frequencies across dimensions (qualitative or quantitative)  would help in understanding the dimensions of temporal variation in greater detail. Especially, are their dimensions with small frequencies? What do they signify because this suggests periodic oscillations.\n\n-- Overall, the paper could benefit from deeper engagement with downstream applications or findings from the temporal embeddings. How exactly are they useful or important?\n\n-- typos: line 166 y(t) missing the i index, line 140, algorithm 1-lines 2 and 3 overload the variable t.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors claim that their work would have no societal impact. I agree to a great extent because the contribution is algorithm oriented and the paper does not touch on potential applications or deployment. \n\nThat said, since the space is provided for reflection on potential negative impact of this work, I think a discussion on the impact of source data used for learning embeddings on the correlations induced by the embeddings would be helpful in more deeply understanding the paper's underlying goal. Documents across several time periods are not all equal: they hide complex decisions around archival choices, bias in publication, etc. Acknowledging this would avoid blind spots in any future studies that use this work to test/propose social science theories. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}, {"review_id": "2CgwbkHNUng", "reviewer": "Reviewer_oVhR", "summary": "\nThis paper presents a new approach to modeling word embedding change over time: each word $w_i$ is represented as a function $g_i(t) : N \\rightarrow R^D$ that maps a timestamp $t \\in \\mathbb{N}$ to an embedding $e_{it} \\in \\mathbb{R}^D$. The function $g_i(t)$ can then be parameterized in a variety of ways; the paper focuses primarily on sinusoidal parameterizations. The paper motivates this approach via the Weierstrass theorem, from which it follows that a real function on the interval [a,b] can be well approximated by a trigonometric polynomial.\n\nThe key contribution of this paper is that it breaks out of the typical paradigm that has been used for diacronic word embeddings: i.e., that a separate word embedding $e_{it}$ is explicitly learned for each word $w_i$ and each timestamp $t$. Doing so comes with several drawbacks that this paper addresses: (1) that an alignment between timestamps must be learned (2) that the alignments are (usually) only dependent in a pairwise Markovian fashion, thus missing out on longer time effects (3) that space required is high.\n\nThis paper evaluates its method with different parametric functions (e.g. linear, word/time independent, and various sinusoids) on several benchmarks established in prior work for the evaluation of diacrhonic word embeddings and attains (mostly) better results than strong baselines from prior work. \n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The paper was very clearly written and packs a lot into its narrative. In particular, S2.2 and S4.2 exemplify the paper's nack for explaining dense technical ideas in a concise yet clear way.\n- This paper is obvious in retrospect in the best possible way. From the title, one might think that this paper sets out to accomplish something difficult (modeling words as functions), but reveals quickly that the idea itself is extremely simple (i.e. simply parameterizing the functions in natural ways and training by backprop). The beauty of its contribution is that no one seems to have thought of doing so before.\n- The experiments suggest that this method works empirically better than prior work despite using dramatically fewer parameters.\n\nWeaknesses:\n- The most significant problem with the paper is the lack of nuance in describing word change over time. For example, the initial motivating example in the Introduction is an example of a new sense of a word appearing that overtakes the previous senses of the word in popularity. Such a change is not well-captured by a smoothly varying function over time at all! In fact, it would be better captured by a model which allows the introduction of additional senses over time that (may or may not) correlate with past senses of the same word. The paper makes no attempt to explain this mismatch between motivating example and method. (Having said this, such an egregious error could be fixed in the writing. Doing so would require re-motivating this approach through the lens of words whose change over time does NOT occur do to the introduction of a new sense, but rather through the more gradual change of their use over time in relation to other words.) Why is this so important? Because the paper does not at present adequately explain *why* it might be outperforming other methods. \n- The main piece that is missing from this paper is additional analysis into the types of functions that are being learned. (At present the only hint of analysis given is a very brief anecdote about how nearest neighbors of specific words vary over time.) What do the learned functions look like? Are there periodic aspects to them or do they stretch out the sinusoids in order to avoid periodic looking functions? Can we learn anything about word change by investing the shape of the functions across larger groups of words?\n- The paper never mentions other possible applications of the idea behind representing words as functions. For example, instead of modeling how words vary over time, one could add another dimension (e.g. genre) and explore how words vary over that dimension.\n- No attempt is made at bridging the gap between the Weierstrass theorem + corralaries, which applies to functions defined on closed intervals, and the true problem of learning word embeddings on open intervals. \n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No -- see the first \"weakness\" mentioned above, which highlights an important limitation of this work.\n", "review_text": "", "extra_scores": {"ethical_concerns": "None.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1.5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper was thoroughly vetted by reviewers who were able to confirm its novelty, limitations (some experimental comparisons would have better contextualised the results), the general importance of the problem (learning diachronic word representations) both in terms of direct applications, and the technical appropriateness of the solution. Like early work championed in ML venues such as LDA that went on to have an important impact in application areas, this is a serious technical contribution that will have an long afterlife in diachronic sociolinguistics.
Decision: Accept (Poster)