Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes a new paradigm for modeling temporal word embeddings. Instead of the commonly prevalent approaches that train static embeddings for documents belonging to each timestamp followed by an alignment step, this work attempts to learn embeddings as a continuous function of time. This work proposes function parametrizations that use sinusoidal polynomials which can model arbitrary continuous functions. The approach is empirically compared against relevant baselines on time-aware clustering, temporal analogy, and semantic change detection.
main_review: Positives:

-- The proposed approach convincingly outperforms most of the baselines while being competitive with dynamic word embeddings. The various ablation of the proposed approach also highlight that modeling frequency parameters are more useful than explicitly modeling phase parameters.

-- The proposed approach is a natural way of thinking about temporal embeddings and is a logical succession to the traditional temporal word embeddings.

Negatives:

-- The loss function seems to work but it seems heuristics-oriented and the motivation behind the loss is weak. What is the motivation behind using both time-sensitive f(.) and compass embeddings h(.) in the loss? Does it relate to any other probabilistic objective functions?

-- In table 4, the proposed approaches trail by compass aligned word2vec by a significant margin on test1. While the prposed approach outperforms the same baselines appreciably on test 2 in table 5. Hence, the proposed explantation of compass embeddings using word2vec implementation is unsatisfactory. More ablative and qualitative analysis focusing on differences between test2 and test1 would shed more light on the differences in behavior of the proposed approach and compass aligned embeddings.

-- Analysis of learned frequencies across dimensions (qualitative or quantitative)  would help in understanding the dimensions of temporal variation in greater detail. Especially, are their dimensions with small frequencies? What do they signify because this suggests periodic oscillations.

-- Overall, the paper could benefit from deeper engagement with downstream applications or findings from the temporal embeddings. How exactly are they useful or important?

-- typos: line 166 y(t) missing the i index, line 140, algorithm 1-lines 2 and 3 overload the variable t.

limitations_and_societal_impact: The authors claim that their work would have no societal impact. I agree to a great extent because the contribution is algorithm oriented and the paper does not touch on potential applications or deployment. 

That said, since the space is provided for reflection on potential negative impact of this work, I think a discussion on the impact of source data used for learning embeddings on the correlations induced by the embeddings would be helpful in more deeply understanding the paper's underlying goal. Documents across several time periods are not all equal: they hide complex decisions around archival choices, bias in publication, etc. Acknowledging this would avoid blind spots in any future studies that use this work to test/propose social science theories.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposed a new paradigm to learn diachronic word representations, which models words as functions of time, so that the word meaning in a different time could be correlated and evolves within a continuous process. The paper adopted various functions such as linear function and sinusoidal functions to approximate the word meaning evolution in the context of distributed representations. The experimental results show that this approach is promising in the time-aware NLP tasks like time-aware word clustering, temporal analogy, and semantic change detection.
main_review: Generally, the topics of the paper and the idea of modeling words as functions of time are super interesting. The paper is easy to follow until section 4.1, before which all the motivations are clear and straightforward. My main concern is why not use a neural network to learn the approximation in Eq. 3? E.g., adding a time embedding to a distributed model. I suggest that a more convincing motivation behind the Polynominal approximation should be provided.

Strengths:
* The paper provided a really good background section, which helps the reader better understand the problem settings and related work.
* The experiments includes lots of time-aware tasks that validates the benefits of the methods.

Weaknesses
* Word2Fun does not improve significantly compared to Compass aligned Word2Vec in Table 4 and Table 5.
* The motivation of using polynomials is not clear.

Questions:
* Did you consider polysemous words at the same timestamp?
limitations_and_societal_impact: Please see the main review for further details.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: 
This paper presents a new approach to modeling word embedding change over time: each word $w_i$ is represented as a function $g_i(t) : N \rightarrow R^D$ that maps a timestamp $t \in \mathbb{N}$ to an embedding $e_{it} \in \mathbb{R}^D$. The function $g_i(t)$ can then be parameterized in a variety of ways; the paper focuses primarily on sinusoidal parameterizations. The paper motivates this approach via the Weierstrass theorem, from which it follows that a real function on the interval [a,b] can be well approximated by a trigonometric polynomial.

The key contribution of this paper is that it breaks out of the typical paradigm that has been used for diacronic word embeddings: i.e., that a separate word embedding $e_{it}$ is explicitly learned for each word $w_i$ and each timestamp $t$. Doing so comes with several drawbacks that this paper addresses: (1) that an alignment between timestamps must be learned (2) that the alignments are (usually) only dependent in a pairwise Markovian fashion, thus missing out on longer time effects (3) that space required is high.

This paper evaluates its method with different parametric functions (e.g. linear, word/time independent, and various sinusoids) on several benchmarks established in prior work for the evaluation of diacrhonic word embeddings and attains (mostly) better results than strong baselines from prior work. 

main_review: Strengths:
- The paper was very clearly written and packs a lot into its narrative. In particular, S2.2 and S4.2 exemplify the paper's nack for explaining dense technical ideas in a concise yet clear way.
- This paper is obvious in retrospect in the best possible way. From the title, one might think that this paper sets out to accomplish something difficult (modeling words as functions), but reveals quickly that the idea itself is extremely simple (i.e. simply parameterizing the functions in natural ways and training by backprop). The beauty of its contribution is that no one seems to have thought of doing so before.
- The experiments suggest that this method works empirically better than prior work despite using dramatically fewer parameters.

Weaknesses:
- The most significant problem with the paper is the lack of nuance in describing word change over time. For example, the initial motivating example in the Introduction is an example of a new sense of a word appearing that overtakes the previous senses of the word in popularity. Such a change is not well-captured by a smoothly varying function over time at all! In fact, it would be better captured by a model which allows the introduction of additional senses over time that (may or may not) correlate with past senses of the same word. The paper makes no attempt to explain this mismatch between motivating example and method. (Having said this, such an egregious error could be fixed in the writing. Doing so would require re-motivating this approach through the lens of words whose change over time does NOT occur do to the introduction of a new sense, but rather through the more gradual change of their use over time in relation to other words.) Why is this so important? Because the paper does not at present adequately explain *why* it might be outperforming other methods. 
- The main piece that is missing from this paper is additional analysis into the types of functions that are being learned. (At present the only hint of analysis given is a very brief anecdote about how nearest neighbors of specific words vary over time.) What do the learned functions look like? Are there periodic aspects to them or do they stretch out the sinusoids in order to avoid periodic looking functions? Can we learn anything about word change by investing the shape of the functions across larger groups of words?
- The paper never mentions other possible applications of the idea behind representing words as functions. For example, instead of modeling how words vary over time, one could add another dimension (e.g. genre) and explore how words vary over that dimension.
- No attempt is made at bridging the gap between the Weierstrass theorem + corralaries, which applies to functions defined on closed intervals, and the true problem of learning word embeddings on open intervals. 


limitations_and_societal_impact: No -- see the first "weakness" mentioned above, which highlights an important limitation of this work.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This submission addresses the problem of learning diachronic
word representations. A new method is proposed that models
words as functions and does not rely (it is claimed) on
learning representations for discrete time intervals first
and then postprocessing them.  The new method is potentially
more powerful than prior work since it can in theory
approximate any continuous process.  It uses fewer
parameters, can model longterm processes and has improved
time complexity.  Good performance on a number of tasks is
demonstrated.

main_review: Current methods for learning diachronic representations all
have some drawbacks. So it is great to see this new
proposal, which is in principle more powerful than previous
work (since it can model any continuous function) and seems
to be more sample-efficient (by not partitioning the
overall learning problem into separate small learning
problems first).

However, I have several concerns about the details of the
paper, which are discussed below. Unless these concerns are
addressed, I would lean against accepting the paper.

DISCRETE VS CONTINUOUS MODELS

I'm not sure I understood the claim that all prior work is
limited to comparing consecutive time slices through a
Markov process. Vector space alignment approaches usually
align all time slices to a single designated vector
space. This does not impose a Markovian structure on the
representations. 

On lines 59 to 62, an argument is made that it doesn't
matter whether you model the process as discrete or
continuous. But Equation (2) is discrete in time. Surely,
the claim is not that how you discretize time for Equation
(2) does not matter? Could you say more about what your
discretization is? (e.g., how many time slices you use)

PRIOR WORK

There is at least one other approach that is not based on a
time-stamped corpus:

https://aclanthology.org/N18-1044/

This paper should be discussed and compared against.

TIME COMPLEXITY

The argument that O(TVD) gives you better complexity that
O(kVD) seems weak to me. T is usually relatively
small. Maybe this could be backed up by giving examples
where the number of time slices is large?

MODELING OF LONGTERM PROCESSES

"do not consider meaning evolution in longer timespan"
(lines 6/7)

It would be an interesting claim to say that there are such
longer timespan meaning evolutions that the proposed method
can model whereas prior work cannot.

However, I find this unconvincing without any analysis. It
seems to me that all prior work can model simple longer
timespan meaning evolutions, e.g., if they are simple linear
evolutions. So the question is not so much: can you model
longer timespan evolutions, but rather: do *complex* longer
timespan evolutions occur in reality? The paper lacks
analysis relevant to this question.

MODELING "POWER" OF PROPOSED METHOD

The proposed method can in principle provide a more
powerful modeling framework than prior work. However, there
is no analysis showing that it does anything interesting
over and above prior work in my opinion.

The authors do not seem to be very knowledgeable in the
linguistic details of diachronic change. A good reference to
refer to would be this:

Peter Koch. 2016. Meaning change and semantic shifts. In
Päivi Juvonen and Maria
Koptjevskaja-Tamm, editors, The lexical typology of semantic
shifts, pages 21–66. De Gruyter,
Berlin.

Koch proposes that diachronic change consists of a process
where a word with the initial meaning M1 acquires a second
meaning M2. M2 initially may have frequency close to 0, but
then increases over time. So one theory of diachronic change
is as a process of a change in frequency distribution of
underlying senses.

One could derive many interesting hypotheses from this
theory and test them using word2fun.

For example, are there any *individual* words that mix
shortterm and longterm evolutionary components in their
meaning change? If it's a simple frequency distribution
change, then maybe not.

Does an individual word show different speeds in meaning
change over time? There should be examples of that, e.g., a
very slow increase of M2 initially and then a quick fading
of M1 later.

Do different words have different time scales with respect
to meaning change? I would suspect yes, and the model (if it
actually lives up to its promise and works as intended)
should be able to show this.

Without any analysis of what the model is actually learning
(beyond some performance numbers), it is not possible for me
to determine whether it is doing something interesting
beyond prior work or not.

EVALUATION

My understanding of the semantic change evaluation is that a
non-standard setup is chosen, so it is difficult to compare
model performance to prior work. I would find it necessary
that experiments are run that guarantee direct comparability
to Semeval-2020.

MINOR POINTS

"Because of the equivalence between prediction-based word
embedding and matrix factorization[12]"

It is generally believed that the proof in [12] is not
correct, so the equivalence should not be presented as
truth, but as a conjecture.

The notion of compass (line 129) is not sufficiently
introduced and explained.







RESPONSE TO AUTHOR RESPONSE AND REVIEWER DISCUSSION


I would like to thank the reviewers for responding to my
review!

I'm not very familiar with the SemEval dataset, but what you
write makes sense. It is still unfortunate that there is no
direct comparison to prior published numbers.

> We believe that the revised paper has been improved,
> unfortunately, the submit system does not allow us to
> update the paper (as ICLR did).

This is indeed a big issue. I could imagine that I would
view the paper more favorably if I could read the updated
version, but since that's not possible, I don't feel
confident about upgrading my review.

It's a particularly big issue for this:

> Analysis of what the model is actually learning We will add
> a new section and Appendix B to quantitatively show and
> visualize what the model is actually learning.

Since I cannot look at this, I don't feel comfortable
allowing it to influence my recommendation.

> The polysemous (multiple meanings for a single word) issue
> cannot easily be solved by typical word vectors since they
> adopt a paradigm of "one vector per word", one may refer to
> "contextualized word embedding" like ELMO, BERT for this
> issue.

This is a longer discussion, but it is actually possible to
analyze single-vectors for polysemy. For example, if sense A
dominates in the first period and sense B dominates in the
last period, then the first single-vector will be close to
words with meanings similar to sense A and the last
single-vector will be close to words with meanings similar
to sense B.

I don't feel my concerns about Rosenfeld & ErK and about
analysis were addressed. I concur with my fellow reviewers
on these points:

oVhR: I would worry that if this paper were accepted they
might continue to downplay prior work in the same way,
rather than appropriately attributing the main idea of
modeling words as functions to R&E'18 as deserved.

rTwC: While, the analysis that finds that most of the words
do not complete a full period is interesting, I agree with
other reviewers that such analysis regarding several aspects
of the diachronic change is crucial to better understanding
of the effectiveness of the approach. In fact, in response
to R6eKb, the authors' proposed metric of correlation b/w
temporal change via analyzing gradients (not in the paper)
exhibits low correlation values. While this doesn't
necessarily mean that the proposed approach doesn't work,
this certainly underscores the importance of a more thorough
analysis to understand the behavior of the proposed
approach.



limitations_and_societal_impact: Yes.
review_title: null

Meta-review: This paper was thoroughly vetted by reviewers who were able to confirm its novelty, limitations (some experimental comparisons would have better contextualised the results), the general importance of the problem (learning diachronic word representations) both in terms of direct applications, and the technical appropriateness of the solution. Like early work championed in ML venues such as LDA that went on to have an important impact in application areas, this is a serious technical contribution that will have an long afterlife in diachronic sociolinguistics.
Decision: Accept (Poster)