SlowGuess/NAACL_2022_OCR huggingface dataset:
Guy D. Rosin and Kira Radinsky
Technion - Israel Institute of Technology, Haifa, Israel
{guyrosin, kirar} @cs.technion.ac.il
Pretrained language models based on the transformer architecture have shown great success in NLP. Textual training data often comes from the web and is thus tagged with time-specific information, but most language models ignore this information. They are trained on the textual data alone, limiting their ability to generalize temporally. In this work, we extend the key component of the transformer architecture, i.e., the self-attention mechanism, and propose temporal attention—a time-aware self-attention mechanism. Temporal attention can be applied to any transformer model and requires the input texts to be accompanied with their relevant time points. It allows the transformer to capture this temporal information and create time-specific contextualized word representations. We leverage these representations for the task of semantic change detection; we apply our proposed mechanism to BERT and experiment on three datasets in different languages (English, German, and Latin) that also vary in time, size, and genre. Our proposed model achieves state-of-the-art results on all the datasets.
Language models (LMs) are usually pretrained on corpora derived from a snapshot of the web crawled at a specific moment in time (Devlin et al., 2019; Liu et al., 2019). But our language is constantly evolving; new words are created, meanings and word usages change. For instance, the COVID-19 pandemic has caused significant changes to our language; consider the new video-related sense of "Zoom" and the new senses recently associated with the word "vaccine".
The "static" nature of existing LMs makes them unaware of time, and in particular unaware of language changes that occur over time. This prevents such models from adapting to time and generalizing temporally (Röttger and Pierrehumbert, 2021;
Lazaridou et al., 2021; Hombaiah et al., 2021; Dhingra et al., 2022; Agarwal and Nenkova, 2021; Loureiro et al., 2022), abilities that were shown to be important for many tasks in NLP and Information Retrieval (Kanhabua and Anand, 2016; Rosin et al., 2017; Huang and Paul, 2019; Röttger and Pierrehumbert, 2021; Savov et al., 2021). Recently, to create time-aware models, the NLP community has started to use time as a feature in training and fine-tuning language models (Dhingra et al., 2022; Rosin et al., 2022). These two studies achieve this by concatenating a time token to the text sequence before training the models. The former was concerned with temporal question answering, whereas the latter—with semantic change detection and sentence time prediction. In this work, we introduce a new methodology to create time-aware language models and experiment on the task of semantic change detection.
At the heart of the transformer architecture is the self-attention mechanism (Vaswani et al., 2017). This mechanism allows the transformer to capture the complex relationships between words by relating them to each other multiple times. An attention weight has a clear meaning: how much a particular word will be weighted when computing the next representation for the current word (Clark et al., 2019). This mechanism also enables the above-mentioned temporal models (Dhingra et al., 2022; Rosin et al., 2022) to work; by concatenating time-specific tokens to the text sequences, the self-attention mechanism would compute the relationships between them and the original tokens in the texts, effectively making the output embeddings time-aware (as the output embeddings will depend on the concatenated time tokens).
In this work, instead of changing the text sequences as in prior work, we modify the model itself and specifically the attention mechanism to make it time-aware. We propose a time-aware self-attention mechanism that is an extension of the
Findings of the Association for Computational Linguistics: NAACL 2022, pages 1498 - 1508 July 10-15, 2022 ©2022 Association for Computational Linguistics
self-attention mechanism of the transformer. It considers the time the text sequences (or documents) were written when computing attention scores. As described above, self-attention captures relationships between words. We want to condition these relationships on time. By adding a time matrix as an additional input to the self-attention (along with the standard query, key, and value matrices), we condition the attention weights on the time. In other words, the adapted mechanism also considers the time when calculating the weights of each word. We refer to this adapted attention as Temporal Attention (Section 3.2). See Figure 1 for an illustration of our proposed mechanism.
We experiment on the task of semantic change detection — the task of identifying which words undergo semantic changes and to what extent. Semantic change detection methods are used in historical linguistics and digital humanities to study the evolution of word meaning over time and in different domains (Kutuzov et al., 2018). Most existing contextual methods detect changes by first embedding the target words in each time point and then either aggregating them to create a time-specific embedding (Martinc et al., 2020a), or computing a cluster of the embeddings for each time (Giulianelli et al., 2020; Martinc et al., 2020b; Montariol et al., 2021; Laicher et al., 2021). The embeddings or clusters are compared to estimate the degree of change between different times. We experiment with several diverse datasets in terms of time, language, size, and genre. Our empirical results show that our model outperforms state-of-the-art methods (Schlechtweg et al., 2019; Martinc et al., 2020a; Montariol et al., 2021; Rosin et al., 2022).
Our contributions are threefold: (1) We introduce a time-aware self-attention mechanism as an extension of the original mechanism of the transformer. The proposed mechanism considers the time the text sequences were written. The time is considered during the computation of attention scores, thus allowing to create time-specific contextualized word representations; (2) We conduct evaluations on the task of semantic change detection and reach state-of-the-art performance on three diverse datasets in terms of time, language, size, and genre; (3) We contribute our code and trained models to the community for further research.
Figure 1: High-level illustration of our proposed temporal attention mechanism.
2.1 Temporal Language Models
Several recent studies have explored and evaluated the generalization ability of language models to time (Röttger and Pierrehumbert, 2021; Lazaridou et al., 2021; Agarwal and Nenkova, 2021; Hofmann et al., 2021; Loureiro et al., 2022). To better handle continuously evolving web content, Hombaiah et al. (2021) performed incremental training. Dhingra et al. (2022) experimented with temporal language models for question answering. They focused on temporally-scoped facts and showed that conditioning temporal language models on the temporal context of textual data improves memorization of facts. Rosin et al. (2022) similarly concatenated time tokens to text sequences and introduced the concept of time masking (specific masking for the added time tokens). They focused on two temporal tasks: semantic change detection and sentence time prediction. Others focused on document classification by using word-level temporal embeddings (Huang and Paul, 2019) and adapting pretrained BERT models to domain and time (Röttger and Pierrehumbert, 2021). Recently, Hofmann et al. (2021) jointly modeled temporal and social information by changing the architecture of BERT and connecting embeddings of adjacent time points via a latent Gaussian process.
In this work, we create a temporal LM by adapting the transformer's self-attention mechanism to time. The model receives each text sequence along with its writing time and uses both as input to the temporal attention mechanism. As a result, the model creates time-specific contextualized word embeddings.
2.2 Semantic Change Detection
Semantic change detection is the task of identifying words that change meaning over time (Kutuzov et al., 2018; Tahmasebi et al., 2018). This task is often addressed using time-aware word representations that are learned from time-annotated corpora
https://github.com/guyrosin/temporal attention
and then compared between different time points (Jatowt and Duh, 2014; Kim et al., 2014; Kulkarni et al., 2015; Hamilton et al., 2016; Dubossarsky et al., 2019; Del Tredici et al., 2019). Gonen et al. (2020) used a simple nearest-neighbors-based approach to detect semantically-changed words. Others learned time-aware embeddings simultaneously over all time points to resolve the alignment problem, by regularization (Yao et al., 2018), modeling word usage as a function of time (Rosenfeld and Erk, 2018), Bayesian skip-gram (Bamler and Mandt, 2017), or exponential family embeddings (Rudolph and Blei, 2018).
All aforementioned methods limit the representation of each word to a single meaning, ignoring the ambiguity in language and limiting their sensitivity. Recent contextualized models (e.g., BERT (Devlin et al., 2019)) overcome this limitation by taking sentential context into account when inferring word token representations. Such models were applied to diachronic semantic change detection, where most detect changes by creating time-specific embeddings or computing a cluster of the embeddings for each time, and then comparing these embeddings or clusters to estimate the degree of change between different times (Hu et al., 2019; Martinc et al., 2020b,a; Giulianielli et al., 2020; Laicher et al., 2021; Montariol et al., 2021). Recently, Rosin et al. (2022) suggested another approach of detecting semantic change through predicting the writing time of sentences. In our work, we use language models to create time-specific word representations and compare them to detect semantic change. While the above studies used language models as is, we modify their inner workings to make them time-aware by adapting the self-attention mechanism to time.
Our model adopts a multi-layer bidirectional transformer (Vaswani et al., 2017). It treats words in the document as input tokens and computes a representation for each token. Formally, given a sequence of  $n$  words  $w_{1}, w_{2}, \ldots, w_{n}$ , the transformer computes  $D$ -dimensional word representations  $x_{1}, x_{2}, \ldots, x_{n} \in \mathbb{R}^{D}$ .
The self-attention mechanism is the foundation of the transformer (Vaswani et al., 2017). It relates tokens to each other based on the attention score
Figure 2: Illustration of our proposed temporal attention mechanism.
between each pair of tokens. In practice, the attention function is computed on a set of tokens simultaneously; our input sequence is packed together into a matrix  $X \in \mathbb{R}^{n \times D}$ , in which each row  $i$  corresponds to a word representation  $x_{i}$  in the input sentence. We denote three trainable weight matrices by  $W_{Q}, W_{K}, W_{V} \in \mathbb{R}^{D \times d_{k}}$ . We then create three distinct representations, i.e., query, key, and value:  $Q = XW_{Q}, K = XW_{K}, V = XW_{V}$ , respectively, where  $Q, K, V \in \mathbb{R}^{n \times d_{k}}$ .
An attention function can be described as mapping a query and a set of key-value pairs to an output, where the query, keys, values, and outputs are all vectors. The output is computed as a weighted sum of the values, where the weight assigned to each value is determined by the dot product of the query with all the keys:
$$
\operatorname {A t t e n t i o n} (Q, K, V) = \operatorname {s o f t m a x} \left(\frac {Q K ^ {\intercal}}{\sqrt {d _ {k}}}\right) V \tag {1}
$$
3.2 Temporal Attention
We now describe the temporal attention mechanism. In the temporal setting, similarly to the vocabulary of the model, our model has a vocabulary of time points. Theoretically, each token in an input sequence could have its own time point, but we simplify and assume the most common case where text sequences always refer to a single time point  $t$ . Given a sequence of  $n$  words  $w_{1},w_{2},\ldots ,w_{n}$  and its corresponding time point  $t$ , our model computes  $D$ -dimensional time-specific word representations  $x_{1}^{t},x_{2}^{t},\dots,x_{n}^{t}$ , where  $x_{i}^{t}\in \mathbb{R}^{D}$ . As a by-product,
2Our mechanism also supports the setting where different tokens in a sequence are associated with different time points.
we also compute  $D$ -dimensional time representations for the time points. Now, similarly to the input matrix  $X$  (Section 3.1), we define an embedding matrix  $X^t \in \mathbb{R}^{n \times D}$  where each row  $i$  contains the embedding vector of  $x_i$ 's time point.
To incorporate time in the attention mechanism, we use an additional trainable weight matrix  $W_{T} \in \mathbb{R}^{D \times d_{k}}$  and create its corresponding representation matrix  $T = X^{t}W_{T}$ . Note  $T \in \mathbb{R}^{n \times d_{k}}$ , i.e., its dimensions are the same as the key, query, and value matrices.
To calculate the attention scores, we multiply the query matrix by the time matrix and then multiply by its transposed matrix, to keep the dimensions intact. We then divide by the time matrix's norm, to avoid getting too large values. Formally, we define temporal attention by:
$$
\text {T e m p o r a l A t t e n t i o n} (Q, K, V, T) =
$$
$$
\operatorname {s o f t m a x} \left(\frac {Q \frac {T ^ {\intercal} T}{\| T \|} K ^ {\intercal}}{\sqrt {d _ {k}}}\right) V \tag {2}
$$
Intuitively, by multiplying the query by the time, the attention weights are now conditioned on the time, i.e., they are time-dependent.
Temporal attention can be used together with other, existing temporal language models, such as (Rosin et al., 2022; Dhingra et al., 2022). In these two models, a time-specific token is prepended to each sentence. In comparison to those methods, our approach does not require changing the input text, as it only modifies the attention mechanism of the language model. We further discuss and compare the two methods in Section 3.3.
The temporal attention mechanism requires each input text to be accompanied with a time point. There are no constraints on these time points, i.e., the mechanism is agnostic to the time granularity and the number of time points in the model.
We now theoretically analyze the temporal attention mechanism more deeply and compare it to existing time concatenation methods (Dhingra et al., 2022; Rosin et al., 2022). We omit the scaling factor  $\sqrt{d_k}$  for readability.
We denote the row vectors of the matrices  $Q$ ,  $K$ ,  $V$ , and  $T$  by  $q_i$ ,  $k_i$ ,  $v_i$ , and  $t_i$ , respectively.
The attention head computes attention weights  $\alpha$  between all pairs of words as softmax-normalized dot products between the query and key vectors:
$$
\alpha_ {i j} = \operatorname {s o f t m a x} \left(q _ {i} k _ {j} ^ {\intercal}\right) \tag {3}
$$
where  $i,j\in \{1,\ldots ,n\}$
The output  $y_{i}$  of the attention head is a weighted sum of the value vectors:
$$
y _ {i} = \sum_ {j = 1} ^ {n} \alpha_ {i j} v _ {j} = \sum_ {j = 1} ^ {n} \operatorname {s o f t m a x} \left(q _ {i} k _ {j} ^ {\top}\right) v _ {j} \tag {4}
$$
Baseline models, such as Rosin et al. (2022) and Dhingra et al. (2022), presuppose the text sequence with a time token at index 0, resulting in:
$$
y _ {i} = \sum_ {j = 0} ^ {n} \alpha_ {i j} v _ {j} = \sum_ {j = 1} ^ {n} \alpha_ {i j} v _ {j} + \operatorname {s o f t m a x} \left(q _ {0} k _ {0} ^ {\top}\right) v _ {0} \tag {5}
$$
As we can see, by concatenating the time token, we add query, key, and value vectors for that token, i.e., a time component is added to the weighted sum.
In contrast, by using temporal attention, the attention weights become:
$$
\alpha_ {i j} = \operatorname {s o f t m a x} \left(q _ {i} \frac {t _ {i} t _ {j} ^ {\intercal}}{\| T \|} k _ {j} ^ {\intercal}\right) \tag {6}
$$
The  $i$ -th output vector  $y_{i}$  is computed as:
$$
y _ {i} = \sum_ {j = 1} ^ {n} \alpha_ {i j} v _ {j} = \sum_ {j = 1} ^ {n} \operatorname {s o f t m a x} \left(q _ {i} \frac {t _ {i} t _ {j} ^ {\intercal}}{\| T \|} k _ {j} ^ {\intercal}\right) v _ {j} \tag {7}
$$
Intuitively, we multiply by the vectors of time to scale the attention weight  $\alpha_{ij}$  by time. We observe two main differences between our proposed mechanism and prior work:
1. The time component is more tightly integrated in temporal attention: instead of just adding a time component to the weighted sum, in temporal attention the time component is multiplied by every component in the sum.
2. Temporal attention requires learning an additional weight matrix  $W_{T} \in \mathbb{R}^{D \times d_{k}}$ . In prior work, each input sequence is prepended with a time token, i.e., its length  $n$  is increased by 1. As a result, the temporal attention mechanism consumes more memory (as it has additional  $D \cdot d_{k}$  trainable parameters $^{4}$ ), whereas prior
3Most tokens share the same time point, as noted above, except for special tokens such as padding and masking tokens, to which we associate unique time points.
4When using the standard BERT-base architecture:  $D\cdot$ $d_{k} = 768\cdot 64 = 49,152$
work requires more time to train (as its sequences are longer). From our experiments, the overhead of both methods is negligible compared to the memory consumption and training time of BERT (see analysis in Section 6.3).
4 Semantic Change Detection
In this section, we employ our proposed temporal attention mechanism (Section 3.2) for the task of semantic change detection (Kutuzov et al., 2018; Tahmasebi et al., 2018). The ability to detect and quantify semantic changes is important to lexicography, linguistics, and is a basic component in many NLP tasks. For example, search in temporal corpora, historical sentiment analysis, and understanding historical documents. The objective of this task is to rank a set of target words according to their degree of semantic change between two time points  $t_1$  and  $t_2$ . In this work, we follow the practice of (Martinc et al., 2020a; Rosin et al., 2022) to estimate the semantic change a word underwent and rank the target words based on these estimates.
Given a target word  $w$ , we generate time-specific representations of it and compare them to detect semantic changes. Algorithm 1 formally describes the method. We begin by sampling  $n$  sentences containing  $w$  from each time point  $t \in \{t_1, t_2\}$  (line 3). For each sentence sent, we create a sequence embedding by running it through the temporal attention model (note the model receives as input both sent and  $t$ ) and extracting the model's hidden layers that correspond to  $w$  (lines 5-6). We then choose the last  $h$  hidden layers and average them to get a single vector (line 7). This is the contextual word embedding of  $w$ , denoted by  $v$ . Following, the resulting embeddings are aggregated at the token level and averaged (line 10), in order to create a non-contextual time-specific representation for  $w$  for each time  $t$ , denoted by  $x_t$ . Finally, we estimate the semantic change of  $w$  by measuring the cosine distance (cos_dist) between two time-specific representations of the same token (line 12).
To train and evaluate our models, we use data from the SemEval-2020 Task 1 on Unsupervised Detection of Lexical Semantic Change (Schlechtweg et al., 2020). We use corpora provided by this task
for English, German, and Latin, covering a variety of genres, times, languages, and sizes. They are all long-term: the English and German corpora span two centuries each, and the Latin corpus spans more than 2000 years. The German corpus is much larger than the other two (x7 - x10). Each corpus is genre-balanced, and split into two time points; see Table 1 for their statistics.
Each corpus is accompanied with labeled data for semantic change evaluation. We use the data from Subtask 2 of this task, where the objective is to rank a set of target words according to their degree of semantic change between  $t_1$  and  $t_2$ . The provided data is a set of target words that are either words that changed their meaning(s) (lost or gained a sense) between the two time points, or stable words that did not change their meaning during that time. The target words are balanced for part of speech (POS) and frequency. Each target word was assigned a graded label (between 0 and 1) according to their degree of semantic change (0 means no change, 1 means total change). For the English dataset, we follow (Montariol et al., 2021) and remove POS tags from both the corpus and the evaluation set.
We use the following baseline methods:
Corpus	C1 Source	C1 Time	C1 Tokens	C2 Source	C2 Time	C2 Tokens	Target Words
SemEval-English	CCOHA	1810–1860	6.5M	CCOHA	1960–2010	6.7M	37
SemEval-Latin	LatinISE	-200–0	1.7M	LatinISE	0–2000	9.4M	40
SemEval-German	DTA	1800–1899	70.2M	BZ, ND	1946–1990	72.4M	48
Table 1: Corpora for semantic change detection. Each corpus is split into two time points, denoted by C1 and C2.
1. Schlechtweg et al. (2019) train Skip-gram with Negative Sampling (SGNS) on two time points independently and align the resulting embeddings using Orthogonal Procrustes. They compute the semantic change scores using cosine distance.
2. Gonen et al. (2020) use SGNS embeddings as well. They represent a word in a time point by its top nearest neighbors according to cosine distance. Then, they measure semantic change as the size of intersection between the nearest neighbors lists in the two time points.
3. Martinc et al. (2020a) were one of the first to use BERT for semantic change detection. They create time-specific embeddings of words by averaging token embeddings over sentences in each time point, and then compare them by calculating cosine distance.
4. Montariol et al. (2021) use BERT to create a set of contextual embeddings for each word. They cluster these embeddings and then compare the cluster distributions across time slices using various distance measures. We use their best-performing method for each dataset as reported in the paper, which uses affinity propagation for clustering word embeddings and either Wasserstein or Jensen-Shannon distance as a distance measure between clusters.
5. Rosin et al. (2022) create a time-aware BERT model by preprocessing input texts to concatenate time-specific tokens to them, and then masking these tokens while training. They introduce two methods to measure semantic change, namely temporal-cosine and time-diff. We use their best-performing method as reported in the paper, which is temporal-cosine.
6. "Scaled attention": We present several baselines which are simplified versions of our temporal attention mechanism. Intuitively, our mechanism differentiates between different time points by learning a scaling factor per
each pair of time points (based on the multiplication of learned time vectors; see Section 3.2). In these baselines, we use a constant scaling factor per time point and calculate attention weights using the following formula:
$$
\alpha_ {i j} = \operatorname {s o f t m a x} \left(q _ {i} s _ {i j} k _ {j} ^ {\intercal}\right)
$$
where  $s_{ij}$  is the scaling factor. This scaling method can be seen as a combination of (Martinc et al., 2020a) and our temporal attention method. We present three options for  $s_{ij}$ : (1) Linear scaling. We hypothesize that recent texts should be given more weight, and define  $s_{ij} = index(t_i)$ , where  $index(t_i)$  is the index of the time point  $t_i$  out of all time points  $t_1, \ldots, t_{n_t}$ . (2) Exponential scaling: similarly to linear scaling, but using an exponent:  $s_{ij} = 2^{index(t_i)}$ . (3) Proportional to the number of documents: here we hypothesize that larger corpora should be given more weight, and define  $s_{ij} = \frac{doc\_count(t_i)}{\sum_{k=1}^{n_t} doc\_count(t_k)}$ , where  $doc\_count(t_i)$  is the number of documents in  $t_i$ .
5.3 Our Method