Efficient Continue Training of Temporal Language Model with Structural Information
Authors: Zhaochen Su
Juntao Li
Zikang Zhang
Zihan Zhou
Min Zhang
Venue: EMNLP 2023 Findings (2023)
Keywords: Temporal Generalization
Syntactic Change
Temporal Language Model
Pre-trained Language Model

Abstract: Current language models are mainly trained on 
snap-shots of data gathered at a particular time, which decreases their capability to generalize over time and model language change.
To model the \textit{time} variable, existing works have explored temporal language models (e.g., TempoBERT) by directly incorporating the timestamp into the training process. While effective to some extent, these methods are limited by the superficial temporal information brought by timestamps, which fails to learn the inherent changes of linguistic components. In this paper, we empirically confirm that the performance of pre-trained language models (PLMs) is closely affiliated with syntactically changed tokens. Based on this observation, we propose a simple yet effective method named \textit{\textbf{S}yntax-\textbf{G}uided \textbf{T}emporal \textbf{L}anguage \textbf{M}odel}~(SG-TLM), which could learn the inherent language changes by capturing an intrinsic relationship between the \textit{time} prefix and the tokens with salient syntactic change.
Experiments on two datasets and three tasks demonstrate that our model outperforms existing temporal language models in both memorization and generalization capabilities.
Extensive results further confirm the effectiveness of our approach across different model frameworks, including both encoder-only and decoder-only models (e.g., LLaMA).
Our code is available at \url{https://github.com/zhaochen0110/TempoLM}.

Efficient Continue Training of Temporal Language Model with Structural Information
Zhaochen Su', Juntao Li'; Zikang Zhang', Zihan Zhou', Min Zhang' Institute of Computer Science and Technology, Soochow University, China
DDpparmentt of Chinese Language and Literature, Peking University {suzhaochen@110 10, zi z com; zhouzihan@pku. edu. cn; {ljt, m2,,mimnne minnmang edu. cn
Abstract
Current language models are mainly trained snap-shots of data gathered a particular time, which decreases their capability to gener- alize over time and model language change. To model the time variable, existing works have explored temporal language models (e.g.. Tem- PoBERT) by directly incorporating the times- tamp into the training process. While effective to some extent, these methods are limited by the superficial temporal information brought by timestamps, which fails to learn the inherent changes of linguistic components. this paper, empirically confirm that the performance of pre-trained language models (PLMs) is closely affiliated with syntactically changed tokens. Based on this observation, propose sim- ple yet effective method named Syntax-Guided Guided Temporal Language Model (SG-TLM), which could learn the inherent language changes by capturing an intrinsic relationship between the time prefix and the tokens with salient syn- tactic change. Experiments on two datasets and three tasks demonstrate that our model outperforms existing temporal language mod- els in both memorization and generalization capabilities. Extensive results further con- firm the effectiveness of our approach across different model frameworks, including both encoder-only and decoder-only models (c.g.. LLaMA). Our code is available at https:// gi gith.. wtw.hh..ooooatt... 2han//haaccee1111 (aeeaachhe@11e/Teeeeeeeee 10TTempoLM.
Sea
Shana dokat posso Trup
(wd.sodlee possane juofy Airea)
Figure I: Representative examples syntactic role changes across time. We track the most frequent syntactic roles of the word "oval" and "justify" from 2015 to 2018, where the syntactic role distribution of the word "justify" is stable across years, while the word "oval" varies dramatically at different times.
et al., 2021) and existing words changing in mean- ings and usage (Labov, 2011; Eisenstein al., 2014; Giulianclli et al., 2020; Montanelli and Periti, 2023). This "static" training paradigm prevents PLMs from generalizing over time and modeling language change (Lazaridou et al., 2021; Søgaard et al., 2021; Loureiro et al., 2022).
To make the model training more "dynamic", existing studies have explored temporal language models (TLMs), which model temporality by corporating timestamp directly into representation when pre-training LMs, i.e., TempoBERT (Rosin et al., 2022), TempoT5 (Dhingra et al., 2022). Their methods involve prepending special time token to each sequence in the training data. Through stacking multiple attention layers, each token at dif- ferent positions can capture the temporal informa- tion brought by time. With different time prefixes, PLMs can adaptively compute the corresponding temporal representations. Moreover, TLMs have better generalization' capability over time as stan- dard LMs are unaware of which data is "new" and which is "old" due to the absence of timestamps during the training process (Dhingra et al., 2022). Though the methods mentioned above can cap-
1
Introduction
While Pre-trained Language Models (PLMs) have achieved remarkable success on most of NLP tasks (Qiu et al., 2020), they neglect the time variable due to their training on snap-shots of data collected at a particular point of time (De- vlin et al., 2019; Liu et al., 2019). However, language is constantly evolving and chang- ing with new words being created (Huang al., 2014; Rudolph and Blei, 2018; Amba Hombaiah Juntao li is the Corresponding Author.
'Temporal generalization (Lazaridou et al., 202 refers to a model is trained on data before time I' but is tested after T:

---

ture temporal information a certain degree, they can merely incorporate "superficial" temporal infor- mation provided by the time prefix with the Masked Language Model (MLM) objective. Thus, it is nat- ural to leverage more -emporal-specific informa- tion captured by textual tokens, c.g., utilizing a small set of lexicons with salient lexical semantic change (Hamilton et al., 2016; Giulianelli et al., 2020; Tahmasebia et al., 2021) in the very recent LMLM method (Su et al., 2022). In this paper, we launch a thorough study to explore the effects of different methods for lexicon selection based on statistical patterns or linguistic attributes and find that the distributional change of syntactic roles (Ku- tuzov et al., 2021) is the most effective strategy in temporal-specific lexicon selection. As shown in Figure I, the discrete syntactic role distribution of the word "oval" dramatically changes over time, while the word "justify" is stable across years.
One popular method is prepend a timestamp in different forms to lextual sequence, e.g., "<2015> Sydney defeats Paris by points at Oval." " (Rosin et al., 2022), "year: 2015 text: Sydney defeat Paris by points at Oval." (Dhingra et al., 2022), and utilize MLM objective to capture the temporal in- formation brought by the time prefix. Through interacting each time prefix with the correlated tex- tual tokens equally, temporal information injected into the pre-trained representation, which ignores the diachronic change degree of different tokens, c.g., time-specific tokens and aanesagnottic tokens". Thus, it is natural to enhance existing tem- poral language models by "accurately" injecting time information into these time-specific tokens, i.e., the core of building better temporal language models is to select these tokens with the time at- tribute. Normally, it difficult to directly compute or estimate the diachronic change degree or the time attribute of tokens. Existing works mainly leverage the discrepancy of data across different periods to approximate the diachronic change de- gree of tokens. For instance, Su et al. (2022) mea- sures the statistical distance of token representation across years to select these tokens (i.e., lexicon) with salient lexical semantic change. Though ef- fective, existing semantic-based lexicon selection methods require forwarding all training data with a large-scale language model in the data process stage and neglect structure information within the language. To accelerate the lexicon selection pro- cess and leverage structure information within lan- guages, we explore the potential of syntactic role changes of different tokens in this paper that may benefit from the speed superiority of various syn- tactic parsing tools. We claborate on more details of syntax-based lexicon selection below.
be
Based on the above observations, we propose a Temporal Language Model (SG- TLM), which consists of two masking strategies: Syntax-Guided
Guided Masking (SGM) and a Temporal- A Aware Masking (TAM) strategy. Experimental re- sults demonstrate that our proposed method sig- nificantly improves performance over other TLMs methods on two datasets and three tasks. Exten- sive results further confirm the efficiency of our method than the state-of of-t--rrtt--- lexicon selection solution based on semantic change, remarkable transferability across various model frameworks, and its positive impact adaption future data.
Summary of Contributions: (i) We explore the task of efficient syntax-guided lexicon selection, which is more challenging for static PLMs to pre- dict on the time-stratified data. (ii) We propose a simple yet effective Syntax-Guide Guided Temporal Language Model (SG-TLM). (iii) SG-TLM ex- hibits excellent performance than other TLMs in terms of memorization and generalization for down- stream tasks. (iv) Our method demonstrates supe- rior efficiency than the SOTA solution, exhibiting high transferability across different model frame- works and positive adaptive ability to future data.
2.2 Syntax-Guided Lexicon Selection
We construct "syntax-guided lexicons" based on diachronic differences in syntax. According to Su et al. (2022), we first adopt YAKE! (Campos et al., 2018), a couture-based and unsupervised system to extract the candidate keywords W {wf,
, from the texts D' of time t. Then, utilize off-the-shelf Stanza² (Manning et al., 2014) to automatically parse the syntax in- formation for each sentence in the texts Dt and
2 Preliminary Study
2.1 Temporal Language Model
Previous works have explored temporal language models to enhance the capability PLMs in mod- eling language change and generalizing over time.
'Time-specific tokens refer to these with salient (larger than threshold) lexical semantic change, while the left tokens treated as time-agnostic ones in Su et al. (2022). Phttps: / /gg thub. com/stanfor wtasstafornnnnoo sannfodddd//saaaz

---

count the frequency of syntactic roles for each can- didate word wi and store them in the set R (rf, rs..... ...... )- This set is a collection of dictionaries, with each dictionary representing the syntactic roles and their frequencies, which is
Tundom
SCMIN
structured as follows: rf = {k] 2 "デ where ky represents the syntactic role for word wy in time t, and ut is its frequency. For example, if the can- didate word "oval" has the syntactic roles "amod" and "nmod" in time t with frequencies 150 and 100, respectively, the corresponding dictionary in R' would be r r'eat {amod: 150, nmod: 100). Using these syntactic dictionaries, we create fea- ture vectors d, and dy to represent the syntactic profiles of the candidate words in different periods. The size of the feature vectors di and äe may vary across words since we create separate feature lists for each word, including the corresponding syntac- tic roles. To align the vectors for each time, we pad the vectors with for any missing syntactic roles, Finally, we calculate the cosine distance between ãR and de to measure the difference between the syntactic profiles of the candidate words W We use hyper-paramett parameter k to control the degree of syntactic change, ranking the candidate words We based on their cosine values and selecting the top-k words as the syntax-guided lexicon, which consists of the tokens with significantly changed syntactic roles across different periods. Our proposed lex- icon selection method is much faster than those used in LMLM (Su et al., 2022), in which their computation cost will be discussed in Section 4.4.
2315
2016
2017
2018
Year
Figure 2: Results of the ppl. score. S-DMLM achieves the highest ppl. values among six selection methods.
Methods for Comparison We introduce six ap- proaches to lexicon construction: (i) We first adopt two methods for extracting lexicons: random selec- tion and frequency-awart selection. (ii) Besides, we introduce two upproaches to selecting the lex- icons with salient linguistic changes: lexical se- mantic change, i... LMLM (Su et al., 2022) and syntactic role change, i.e., our proposed syntax- guided method (SMLM) introduced in Section 2.2. (iii) Additionally, we consider words that are de- pendent the tokens with significant diachronic change, identified from the syntactic parsing process?, i.e., the head node in the dependency parsing tree (McDonald al., 2005), and propose two methods to include these words dependent on extracted lexicons based on LMLM and SMLM, named L-DMLM and S-DMLM, respectively.
2.3 Discussions and Observations
The Influence of Lexicon Selection Methods that:
The results are shown in Figure 2. We (i) In the absence of adding dependent words, the ppl. of SMLM is much higher than the other three lexicon selection methods, which indicates that it is more challenging for static PLMs to predict the lex- icons selected from the syntactic perspective. (ii) After adding dependency information, both SMLM and LMLM methods show apparent increase in their ppl. values, i.e., L-DMLM and S-DMLM, suggesting the positive impact of adding dependent words from syntactic parsing in lexicon selection. (iii) Above all. S-DMLM (marked with achieves the highest ppl. values among six methods, which can select diachronic change lexicons that impose the most significant challenge to static LMs.
Section 2.2, we propose a direct and efficient syntax-guided approach for obtaining the lexicons which have undergone significant syntactic change over time. Following Su et al. (2022), we mask the tokens in the selected lexicons and utilize per- plexity (ppl.) as a qualitative measure to compare the influence of different lexicon selection meth- ods on static PLMs. To complete this, build time-stratified corpus from publicly released crawl news* datasets, which contains 1M English news articles for each year between 2014 and 2018. We post-tune the BERTS model with the data from 20146 and evaluate the four testing sets after 2015,
"ht //data. http///attt.ttttt..t..m........t www.tt..rr//ewwws.... www//ewwscrawll/ crawl/ "We initialize parameters from BERT-BASE-UNCASED.
"To climinate the impact of domain divergence (Gururan- gan ct al., 2020) on the performance of testing, we post-tune the uniform BERT with the news data in 2014.
"This allows us to capture more nuanced and subtle tempo- ral information present in the text, which is crucial for tasks that require temporal understanding and knowledge retention.

---

1 Syntux-Guided Lexicon Delection
Syntax-Guided Masking
Temporal-Aware Masking
2015
MLM head
zcomn points
2015
nsubi
root
ubll
head
Sydney
points nand
2016
ubj
<MASK-
Sydney defeats Paris
<MASK-
-MASK>
2017
root obl
MLM head
elipuical
nubb
2018
Trump malks out from the Oval Office compound
<MAKK-
Trump walks out from the
-MASK>
-MASK>
Figure 3: The overall structure of our proposed Guide- Temporal Language Model (SG-TLM). construct the syntax- -uided lexicons across time (0) and post-train with two components: Syntax-Guided Masking (SGM) scheme (2) and Temporal-A
Aware ware Masking (TAM) scheme ($).
3 Syntax-Guided TLM
the sentence additional temporal information. Specifically, given the masking ratio a we pri- oritize masking the words in Wmask' and their corresponding words the dependency relation- ship"! " We randomly mask the other tokens from the sequence if there are no sufficient masking can- didates to meet the required number of masking tokens. Assuming it masks tokens in total and the sequence after masking at time is dy. The optimization objective of SGM can be written by:
The Masked Language Model (MLM) objec- tive (Devlin et al., 2019) is widely-adopted self- supervised training method that involves randomly masking certain percentage of the tokens in text sequence and training model to predict the masked tokens based on their context. Previous TLMs add a timestamp token at the beginning of the input sequence and utilize the MLM objective to predict the random masked tokens based on the context and the timestamp. However, these meth- ods disregard inherent temporal-specific informa- tion provided by lexicons with salient change to- kens. Based on the aforementioned observation we propose Syntax-Guided Guided Temporal Language Model (SG-TLM), which consists of two main com- ponents: a Syntax-Guiii Guided Masking (SGM) scheme and a Temporal-A ware Masking (TAM) method. Our proposed model is illustrated in Figure 3.
m
LSGM =
2 log = wl, d'; (1)
Temporal-Aware Masking (TAM) Unlike pre- vious work (Rosin et al., 2022), we predict masked tokens with salient syntactic role change and time tokens, given the remaining unmasked words within the sequence. Formally, given a sequence {di} at time t, we denote d!")) for its timestamp and prepend the time token t to d, Now the inputs to the model are a se- .d""). Assum-
Syntax-Guided Masking (SGM) We construct the syntax syntax-guided guidedd lexicons based on the distribu- tional change of syntactic role across timestamps. Formally, given the text set D = {di, dz.... d time we first rank the lexicons according to the word's cosine values of syntactic change. Then, we select (I+ E {100, 200, ... 500}) words with relative high scores as the masking candidate words Wmask'. Considering the effectiveness of adding dependent words, we treat the words that are de- pendent by the candidate words Wmask within
quence d; = [t, d ((2)
ing the set of masked token is {d(2)) we pre- diet the time t by the whole input text: p(t)
p(tla(") <MASK>, af ......))))) d.")). As for the gran- ularity of different values can be used according to the use case. this work, we experiment with the granularity of year for the WMT dataset, and a month for the RTC dataset.
"Inn Appendix 4.4, will analyze suitable values for the masking ratio a and word counts k.
"In Section 2.3, we discover that utilizing the distributional change of syntactic roles is most effective strategy when selecting temporal-specific lexicons.
"In Section 4.2, we also explore training the model on the tokens with salient syntactic change, without taking the corresponding dependent words (STLM) into account, and the results are shown in Table (marked with t).
"In Appendix C, we will investigate the influence of pars- ing toolkit's 's performance on our SG-TLM.

---

4 Experiments
April 2017, extracted from the same dataset used for pre-training. Memorization and generalization are tested on (S2017-00 04 S2018-08. 52019-08 and ( respectively. We calculate the F1-score as the testing results for all our experi- ments. More details about the task and our model's training are shown in Appendix A.
4.1 Experimental Setup
Datasets We conduct continual pre-training on two datasets: WMT NEWS CRAWL (WMT) and REDDIT TIME CORPUS (RTC), respectively. The WMT dataset, an open-domain dataset, consists of 4 million news articles published between 2015 and 2018: (D2015, D2016; D2017, D2018). On the other hand, the RTC (Röttger and Pierrchumbert, 2021) is monthly time-stratified dataset from March 2017 February 2020. Unlike the WMT dataset, it specifically focuses on the political do- main, enabling us to explore temporal dynamics within the specific domain. We select three months (D2017-04. 22018 D2019-0s} for pre-training, each containing I million unlabeled comments.
4.2 Baselines
We establish several baselines that encapsulate dif- ferent continual pre-training strategies. Firstly, we consider two naive baselines that do not incorpo- rate timestamps during the pre-training stage: (i) BERT (w/o) (Devlin et al., 2019), which is di- rectly fine-tuned on the downstream task without pre-training. (ii) Uniform involves training the model with mixed pre-trained data. Additionally, we adopt three up-to-date TLMs, which utilize timestamps during pre- pre-training: (iii) TAda (Röttger and Pierrehumbert, 2021), which involves contin- ual pre-training on specific time buckets obtain separate models specialized for different periods. (iv) Temporal (Dhingra et al., 2022) integrates time i as a prefix to the input during pre-training with cemporal-specifit lexicons randomly gener- ated. (v) LMLM (Su et al., 2022) is the SOTA of temporal adaptation, which strengthens PLMs generalization with salient lexical semantic change. We utilize their method of lexicon construction. STLM and SG-TLM are our methods, with the distinction of whether to add dependent words dur- ing the lexicons construction. Appendix B offers additional training details on the compared TLMs.
Evaluation Following the pre-training period, we evaluate the models' memorization (Dhingra et al., 2022) and generalization (Lazaridou et al., 2021) abilities by measuring their performance on downstream ta To evaluate memorization, the model is tested on the same time steps as the pre-training data St... evaluate generalization, the model's performance is measured on future times (ST SI+n), which is invisible during the post-tuning stage.
Sr). To
After continual training on the WMT dataset, two tasks are used for model evaluation: po- litical affiliation classification (POLIAFF) (Luu et al., 2022) and named entity recognition (TwINER) (Rijhwani and Preotiuc-Pietro, 2020). The POLIAFF task involves fine-tuning the model with 10,000 labeled sentences from 2015. Test- ing includes (S2015, S2016; S2017, S2018} for mem- orization and {S2019, S2020 for generalization, with each year containing 2,000 specific sen- tences. For the TWINER task, 2,000 labeled tweets from 2015 are selected for fine-tuning. We evaluate memorization abilities using datasets (S2016- S2018] and generalization capabili- ties using dataset {S2019}. Following pre-training on the RTC dataset, the model is evaluated on the political subreddit prediction (PSP) (Röttger and Pierrehumbert, 2021) task. Specifically, the model is fine-tuned on 20,000 labeled data samples from
4.3 Main Results
Table I presents the results on the WMT dataset. TLMs demonstrate superior performance compared to models that do not consider timestamps. How- ever, previous TLMs such as TAda and Temporal show only marginal improvements over the Uni- form model, indicating limited learned temporality information from the timestamps. Conversely, in- corporating linguistic information into TLM train- ing significantly improves both memorization and generalization. Among the evaluated baselines, SG-TLM achieves the highest average F1 F1-scores on both datasets, i.c., 66.77 and 66.67, highlight- ing the effectiveness of leveraging syntax and de- pendency information within languages. Table 2 presents the results for the RTC dataset. Simi- lar to the previous findings, SG-TLM consistently achieves the best performance.
Phttpss//data. statmt. www.o//g/////ooooooo/./o..es crawl/
13 PPrevious works (Röttger and Pierrehumbert, 2021; Lazari- dou et al., 2021) have discovered that the model's performance deteriorates as the gap between the training and testing time increases. Following their setting, we evaluate the model that was fine-tuned on the oldest time step.

---

POLIAFF
TWINER
Model
2015
2016
2017
2018
2019
2020
Avg.
2016
2017
2018
2019
Avg.
BERT
.1.30
63.80g.s
S2.22g.9
45.00
38,93
55.08
75.82a4
56,98g2
56.160.5
56.99..4
61.48
Uniform
91.33ux
74.87
66.014.6
54.670.7
55.750,s
46.57pg
64.87
77.04p6
59.05y.4
58.52g,
59.07
63.42
+ TAda
90.79p9
74.40u.7
65.33u.4
55.230.
53.19
44.95u.s
63.98
75,50p4
60.69
60.72g.a
60.12o.1
64.25
Temporal
90.8466
73,460.5
65.03p.8
55.39
56.8%
46.410.5
64.67
74,02a3
57,540.7
56.840.3
57.5404
61.49
-LLMM
92.81a7
75.82
64.83p.3
55.460.4
57.12
48.20p.s
65.71
79.34o3
60.75g.4
60.01p.8
60.766.3
65.22
+ +TLLM
9
75.566.5
6p.3
55.71oa
57.76)
49.130.2
66.13
81,6703
60,78g
60.09p.4
60.91o.
65.86
+SG-TLM
93.69g.
76.69u.
66.58u.1
55,94..
57.4
50.32g.a
66.77
81.78,
61.85
61.18..1
61.87ou
66.67
Table I: Results of the WMT dataset: memorization and generalization performance on the POL.AF and TWINER tasks. The timestamp represented in italics are not visible during the post-tuning stage, i.e., 2019 and 2020 in POLIAFF dataset and 2019 in TWINER dataset. We utilize the above testing sets evaluate the models' generalization and the left to test the memorization. STLM (marked with f) and SG- SG-TLM (marked with f) are our methods, the difference is whether to include dependent words during the masking scheme. Our proposed SG-TLM achieves the highest average F1-score on tasks. Each number is the average of runs with different seeds.
RTC
LMLM (Su et al., 2022)
Model
17-04
18-04
19-18
20-01
20-02
The speed of representation
200min (1.0x) 360min (1.0x)
BERT
49.61p.s
41.74,.1
38. 23,s
38.84
39.97p.3
The speed of measuring
Uniform
51,97a.7
42,571.3
39,98pu
40.75pg
40.53
Fl-score on average
65.22 (-)
"TAda
50.62g.6
42.2109
39.47
40.34,.3
SG-TLM (Ours)
Temporal
51.69Lo
43.331,
240.a
12.7
39.500.4
-LMLM
51.91g.9
43.261,1
39.4112
40.94p.
40.79p.5
The speed of parsing
36min (5.5x)
The speed of measuring
2min (180x) 66.67 (+1.45)
+TLLM 1
52.48o.6
43.67n.
39.60p.s
41.21::1
40.52g,9
Fl-score on average
+SG-TLM;
52.89
43.51m4
41.11a7
41.25k.
40.87
Table 2: Results of the RTC dataset: memorization and generalization performance on the PSP task (average of 5 runs). We utilize the set in 20-01 and 20-02 to test generalization and the rest to test memorization. Our proposed SG-TLM achieves the highest Fl-score.
Table 3: Efficiency comparison between LMLM and SG-TLM. SG-TLM a faster and more effective model.
tactic parsing tools rather than large-scale PLMs, resulting in a speedup ratio of 5.5x compared to representation and 180x compared to measuring, Furthermore, SG- TLM outperforms LMLM on the TWINER task by achieving 1.45 higher Fl-score. demonstrating its superior effectiveness.
However, the performance differences among the methods the RTC dataset are relatively mi- nor compared to the WMT dataset, which can be attributed to the shorter time intervals and the rel-
atively stable and slight dynamic temporality of the RTC dataset. Moreover, SG-TLM also shows notable drop in future years due to the inherent uncertainty of future language changes (Lazaridou et al., 2021) in both datasets. In Section 4.5, we will investigate the approaches to refreshing the models as new data arrives and compare the tem- poral adaptation (Röttger and Pierrehumbert, 2021; Luu et al., 2022) performance of different methods.
A Ablation Study To investigate the impacts of dif- ferent components within SG-TLM, we remove individual components from the complete model and observe the resulting performance. The results are shown in Table 4. Notably, excluding the SGM objective leads to the most significant decline in performance, highlighting its pivotal role within the SG-TLM framework. Furthermore, each com- ponent contributes positively to the overall perfor- mance, indicating the utility and significant contri- butions of all SG-TLM components in improving the model's effectiveness.
4.4 Detailed Analysis
Efficiency Comparison Table 3 shows the com- parison of lexicon construction efficiency between LMLM and SG-TLM. LMLM provides fairly complex and time-consuming method to select semantic-based lexicons, i.e., 200 minutes for rep- resenting and 360 minutes for clustering. However, SG-TLM benefits from the speed superiority of syn-
Scale Effects in Performance We also explore whether our proposed SG-TLM would keep ef- fective over random masking when increasing the amount of data. Using the WMT dataset, we suc- cessively expand the training data for both models,

---

POLIAFF
TWINER
Model
2015
2016
2017
2018
2019
2020
Avg.
2016
2017
2018
2019
Avg.
Our model
93.69
76.69
66.58
55.94
57.41
50.32
66.77
81.78
61.85
61.18
61.87
66.67
0- SMLM
92.66
75.08
65.35
53.14
55.61
48. 10
64.98
82.71
59.25
60.52
59.56
65.51
0- TMLM
92.86
76.75
66.02
55.48
57.12
49.20
66.20
78.62
60.37
60.42
60.37
64,95
Table 4: Ablation study: Results the POLIAFF and TWINER task (average of runs). We can discover that removing SGM from the SG-TLM has the most significant impact.
TWINER
Methods
rendom MMLM S-DMLM
2016
2017
2018
2019
Avg.
Uniformaar
77.04
59.05
$8.52
59.07
63.42
62.5
SG-TLMcM
81.78
61.85
61.18
61.87
66.67
62.0
Uniformear
77.18
56.78
55.21
56.8
61.49
Fl-score
SG-TLMMar
83.31
61.71
60.96
61.72
66.93
61.5
Uniformizar
79.12
59.12
58.03
59.32
63.89
61.0
SG-TLM12M
83,74
61.63
60.11
61.94
66.86
60.5
Uniform1ea
81.31
59. 94
60.38
58.96
65.14 14
SG-TLM
84,66
62.57
61.57
61.58
67.60
UniformZon
80.47
60.01
59.40
60.41
65.07
59.5
SG-TLM20M
85.27
62.17
61.64
62.18
67.82
Unitorm
LMLM
SG-TLM
Figure Results of different TLMs' adaptability per- formance the target year (average of 5 runs). The horizontal axis indicates the different TLMs, and the vertical axis shows the Fi-score of the TwINER task. Our proposed SG-TLM outperforms other models with highest Fi-score among the nine settings.
Table 5: Results of scale effects in performance (average of 5 runs). In method names, "4M". "8M". etc., denote the use of million, 8 million, etc., datasets for pre- training. SG-TLM and Uniform represent our proposed method and the random masking strategy, respectively.
multiplying the volume by factors of 2, 3, 4, and 5, The results of this scaling experiment summa- rized in Table 5. It is clear that the SG-TLM consis- tently outperforms the uniform masking approach across all years and data scales, demonstrating the robustness of our approach with increasing data.
alignment In this section, we consider the sce- nario where we already have a trained model on the 2015-18 slices and new data from the 2019 slice. We attempt to update the model by continuing pre- training TLMs on the unlabelled target year data, To compare the adaptability of different TLMs on target year, evaluate their performance the TWINER dataset. Precisely, we fine-tune the adap- tation models on the labeled source year data and then test models on 2019 data. Experiments are conducted adapting three TLMs with three lexi- con construction methods, totaling nine settings. Results are shown in Figure 4.
Jlyeer-Paramede Analysis Considering the cor- relation between the masking ratio and the model's performance, we conduct experiments to explore the most suitable masking ratio and word counts k for the SG-TLM objective. The results shown in Figure 5. Our SG-TLM achieves the best perfor- mance when the masking ratio a is set to 30% and the number candidate words is is 200. To better understand the insights of our presented SG-TLM, we conduct token-level analysis on the selected lexicons in Appendix D.
4.6 Transferability Across PLM Frameworks
To verify the transferability of methods across different model frameworks, implement method in both encoder-only and decoder-only models and utilize random lexicon construction as the bascline for comparison.
4.5 Temporal Adaptation to New Data
I' Appedix E, we will provide further analysis on SG- TLM's adaptability to temporal changes from the perspective token prediction. in previous experiments, we use 2015 as the source year.
Unlike domain adaptation, temporal adapta- tion (Röttger and Pierrehumbert, 2021) updates models with current data to mitigate temporal mis-

---

62.5
12.5
62.0
63.5
21-0000
60.2
66LS
60.5
30.5
60.0
60.5
59.5
410
3040
coondidn
Figure 5: Results of the different masking strategies of SG-TLM (average of runs). The horizontal axis indicates the candidate word counts & and the vertical axis represents the Fl-score of the TWINER task.
RTC
RTC
Model
Method
17-04
18-04
19-08
20-01
20-02
21-02
21-08
22-02
22-08
23-02
BERT
$1.97
42.57
39.98
40.75
40.53
Random
14.81
14.98
15.02
15.29
15.46
SG-TLM
52.89
43.51
41.11
41.25
40.87
SyG.
15.82
16.02
16.37
16.55
16.90
RoBERTa
52.09
42.96
39.22
40.81
40.64
+ SG-TLM
54.01
44.81
40.11
41.44
41.84
Table 7: Comparative perplexity results LLaMA-7B on the RT C dataset using Centactic-Guied (SyG.) and random selection methods.
Table 6: Results of different PLMs on the RTC task (average of runs). Our proposed SG-TLM achieves the highest F1-score on both BERT and RoBERTa.
Related work
Effectiveness Encoder-only PLMs We im- plement our method on two popular encoder-only PLMs, i..., BERT and RoBERTa. As shown Ta- ble 6, we observe a significant improvement each PLM using our SG-TLM. On average, the BERT model improves by 0.77 points, and the RoBERTa model improves by 1.31 points. These findings illustrate the versatility of our proposed SG-TLA SG, enhances the performance of various PLMs by incorporating syntax information.
Temporal Language Model Several works have explored the temporal effects in language mod- els (Huang and Paul, 2018, 2019; Rijhwani and Preotiuc-Pietre 2020; Lazaridou et al., 2021; Sø-
gaard al., 2021; Agarwal and Nenkova, 2022; Loureiro et al., 2022: Cao and Wang. 2022: Cheang et al., 2023). Recently, existing works have investi- gated the temporal language model to model tempo- rality information and generalize over time. Dhin- gra et al. (2022) and Rosin et al. (2022) directly prefix the time token to text sequences and fine-tune time-stratified data. Hofmann al. (2021) and Rosin and Radinsky (2022) modify the structure of the language model to create time-specific contex- tualized word representations. Su et al. (2022) re- cently proposed a remantic-based lexical masking strategy to enhance PLMs' temporal generalization. We extend this work with detailed study using syntactic role changes harness temporal-specific information efficiently.
Effectiveness on Decoder-only PLM conduct experiments on large-scale decoder-only language models, i.e., LLaMA-7B (Touvron et al., 2023). In these experiments, extract the word selection component from our method and com- pare two data selection methods: one based on our Syntax-Guie -uided (SyG.) approach and the other on random selection. The perplexity of LLaMA-7h is evaluated on the 2,000 sentences selected using these two methods. As shown in Table 7, SG-TLM yields higher perplexity than random selection in the RTC 16 This highlights the complex- ity and diversity of our selected data, indicating the effectiveness incorporating syntax into data selection and the potential to enhance temporal capture in large language models.
We also
Diachronic Language Change Diachronic lan- guage change can be mainly divided into semantic change (Kurtyigit et al., 2021; Montanelli and Per- iti, 2023), morphological change (Hare and Elman, 1995; Ji al., 2019; Giulianelli al., 2022), and syntactic change (Kroch, 2001: Seretan, 2011: By- bee, 2017; 7; Merrill et al., 2019). Previous work focused on discovering the words that have un- dergone diachronic change under the supervised
1 "Since the training data for LLaMA is cut off at 2022, we crawl the latest Reddit data from https://files. pushshift.ic io to complete the experiment.

---

settings (Kim al., 2014; Basile et al.; Basile and McGillivray, 2018; Tsakalidis et al., 2019; Kurtyigit et al., 2021). Recently, several works have demonstrated that contextualized word rep- resentations have dynamic representation capabili- ties (Pilehvar and Camacho-Colladinn 2019; Chro- nis and Erk, 2020; Garí Soler and Apidianaki. 2021; Laicher et al., 2021; Qiu and Xu, 2022), which are adopted with unsupervised methods repre- sent, cluster, and differentiate words across differ- ent time periods (Giulianelli al., 2020; Montariol et al., 2021). Our method utilizes a syntax-ban based method to detect the salient change words within the text sequence, making the process more inter- pretable (Merrill et al., 2019; Ryzhova et al.; Kutu- zov et al., 2021). To our best knowledge, this is the first work that incorporates syntactic knowledge into the training of temporal language models.
syntactic features in detecting lexical semantic changes. This prompts us to investigate the com- patibility of our method with the semantic lexi- con solution. Given the lexicon constructed by se- mantic change W/ and syntactic change W,, we conduct three straightforward methods to com- bine the lexicons between Wi and W,: W/ N. WUU,,, Wi Wg. Consistent with the previous experiment, the masking ratio a is 30%, and k is 200. The results are shown in Table 8. Con- trary to our expectations, the combined model's performance is inferior as compared to the orig- inal model, suggesting that this method fails to merge information effectively from multiple di- mensions. In future work, we will explore more suitable methods to integrate semantic and syn- tactic information.
RTC
6 Conclusion
Model
17-04
18-04
19-08
20-01
20-02
In this paper, we enhance the temporal language model from the syntactic perspective and discover that predicting the syntax-guided lexicons is more challenging for static PLMs compared to other methods. Building upon these findings, we pro- pose syntax -uiieed temporal language model (SG-TLM) that incorporates time information into tokens with significant syntactic changes. Our SG- TLM achieves the SOTA performance, reduces computational costs during lexicon construction, and demonstrates excellent transferability to new data and frameworks compared to other baselines.
W:
51.91
43.26
39.41
40.94
40.79
W.
52.89
43.51
41.11
41.25
40.87
W. W.
50.20
42.25
38.60
39.54
39.65
W. W.
50.34
42.32
38.28
40.12
39.99
Wi W.
50.24
42.33
38.66
40.13
39.86
Table 8: The compatibility of our method with semantic lexicon solution. Wi represents lexical semantic solu- tion, while W. represents our syntactic role solution.
Acknowledgement
We would like to thank the anonymous reviewers for their constructive comments. This work was supported by the National Science Foundation of China (NSFC No. 62206194 194) and the Natural Sci- ence Foundation of Jiangsu Province, China (Grant No. BK20220488).
7 Limitation
There are still some limitations in our work which are listed below:
While we introduce a data selection strategy that incorporates syntactic changes identify time-specific sentences, we only conduct prelimi- nary validation of our method's transferability on Large Language Models (LLMs), without involv- ing training and inference stages. Recent studies highlight that LLMs continue to struggle with ef- fective generalization when it comes to emerging data (Wang et al., 2023). As potential solution, our future work aims to integrate our method with in-context learning (Dong al., 2022) to enhance the temporal generalization capabilities of I.LMs.
References
Oshin Agarwal and Ani Nenkova. 2022. Temporal ef- fects on pre-trained models for language processing tasks. Transactions of the Association for Computa- tional Linguistics, 10:904 921.
Spurthi Amba Hombaiah, Tao Chen, Mingyang Zhang. Michael Bendersky, and Marc Najork. 2021. Dy- namic language models for continuously evolving content. In Proceedings of the 27th ACM SIGKDD Conference Knowledge Discovery & Data Mining. pages 2514 25244...
Pierpaolo Basile, Annalina Caputo. Roberta Luisi, and Giovanni Semeraro. Diachronic analysis of the ital- ian language exploiting google ngram.
Recent studies (Kutuzov et al., 2021; Giulianelli et al., 2022) show the effectiveness of utilizing

---

Pierpaolo Basile and Barbara McGillivray. 2018. Ex- ploiting the web for semantic change detection. In In- ternational Conference on Discovery Science, pages 194-208. Springer.
with contextualised word representations. In Pro- ceedings of the 58th Anmual Meeting of the Asso- ciation for Computational Linguistics, pages 3960- 3973.
Joan Bybee. 2017. Mechanisms of change in grammati- cization: The role of frequency. The handbook of historical linguistics, pages 602-623.
Mario Giulianelli, Andrey Kutuzov, and Lidia Pivo- varova. 2022. Do not fire the linguist: Grammatical profiles help language models detect semantic change. In Proceedings of the 3rd Workshop on Computa- tional Approaches Historical Language Change, pages 54-67.
Ricardo Campos, Vitor Mangaravite, Arian Pasquali, Alípio Mário Jorge, Célia Nunes, and Adam Jatowt. 2018. Yake! collection-indeeeeneett automatic key- word extractor. In European Conference on Informa- tion Retrieval, pages 806-810. Springer.
Suchin Gururangan, Ana Marasović. Swabha Swayamdipta. Kyle Lo, Iz Beltagy, Doug Downey, and Noah A Smith. 2020. Don't stop pretraining: Adapt language models to domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8342 8360.
Shuyang Cao and Lu Wang. 2022. prompting for text generation. arXiv:221100266.
Time-aware arXiv preprint
Chi Seng Cheang, Hou Pong Chan, Derek F Wong, Xuebo Liu, Zhaocong Li, Yanming Sun, Shudong Liu, and Lidia S Chao. 2023. Temposum: Evaluating the temporal generalization of abstractive summariza- tion. arXiv preprint 230.:2335.0195..
William I. Hamilton, Jure Leskovec, and Dan Jurafsky. 2016. Diachronic word embeddings reveal statistical laws of semantic change. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1489- 1501.
Gabriella Chronis and Katrin Erk. 2020. When is a bishop not like rook? when it's like rabbi! multi- prototype bert embeddings for estimating semantic relationships. In Proceedings of the 24th Confer- ence on Computational Natural Language Learning, pages 227-244.
Mary Hare and Jeffrey L Elman. 1995, Learning and morphological change. Cognition, 56(1):61-98.
Valentin Hofmann, Janet Pierrehumbert, and Hinrich Schütze. 2021. Dynamic contextualized word em- beddings. In Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the 11th International Joint Conference on Natu- ral Language Processing (Volume I: Long Papers). pages 6970-6984.
Jacob Devlin, Ming-Woi Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understand- ing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technolo- gies, Volume I (Long and Short Papers), pages 4171- 4186.
Minlie Huang, Borui Ye, Yichen Wang. Haiqiang Chen, Junjun Cheng, and Xiaoyan Zhu, 2014. New word de- tection for sentiment analysis. In Proceedings of the 52nd Annual Meeting of the Association for Compu- tational Linguistics (Volume I: Long Papers), pages 531-541.
Bhuwan Dhingra, Jeremy R Cole, Julian Martin Eisenschlos. Daniel Gillick, Jacob Eisenstein, and William W Cohen. 2022. Time-aware language mod- els as temporal knowledge bases. Transactions of the Association for Computational Linguistics, 10:257- 273.
Xiaolei Huang and Michael Paul. 2019, Neural tem- porality adaptation for document classification: Di- achronic word embeddings and domain adaptation models. In Proceedings of the 57th Annual Meet- ing of the Association for Computational Linguistics, pages 4114 111333.3.
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiy- ong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. 2022. A survey for in-context learning. arXiv preprint arXiv:2301.022.
Jacob Eisenstein, Brendan O'Connor, Noah A Smith, and Eric P Xing. 2014. Diffusion of lexical change social media. PloS one, 9(11)e
Xiaolei Huang and Michael Paul. 2018. Examining temporality in document classification. In Proceed- ings of the 56th Annual Meeting of the Association for Computational Linguistics ( (Volume 2: Short Papers). volume 2.
Aina Garí Soler and Marianna Apidianaki. 2021. Let's play mono-poly poly: Bert can reveal words' polysemy level and partitionability into senses. Transactions of the Association for Computational Linguistics, 9:825- 844.
Yatu Ji. Hongxu Hou, Chen Junjie, and Nier Wu. 2019. Improving mongolian congolian-chinese chinese neural machine trans- lation with morphological noise. In Proceedings of the 57th Annual Meeting of the Association for Com- putettionll Linguistics: Studeni Research Workshop. pages 123-129.
Mario Giulianelli, Marco Del Tredici, and Raquel Fer- nández. 2020. Analysing lexical semantic change

---

Yoon Kim, Yi-I Chiu, Kentaro Hanaki, Darshan Hegde, and Slav Petrov. 2014. Temporal analysis of lan- guage through neural language models. ACI. 2014, page 61.
Christopher D Manning, Mihai Surdeanu, John Bauer, Jenny Rose Finkel, Steven Bethard, and David Mc- Closky. 2014. The stanford corenlp natural language processing toolkit. In Proceedings of 52nd annual meeting of the association for computational linguis- tics: system demonstrations, pages 55-60.
Anthony S Kroch. 2001. Syntactic change.
Sinan Kurtyigit, Maike Park, Dominik Schlechtweg. Jonas Kuhn, and Sabine Schulte im Walde. 2021. Lexical semantic change discovery. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 6985 6998.
Ryan McDonald, Fernando Pereira, Kiril Ribarov, and Jan Hajic. 2005. Non-projective dependency pars- ing using spanning tree algorithms. In Proceedings of human language technology conference and con- ference on empirical methods in natural language processing, pages 523-530.
William Merrill, Gigi Stark, and Robert Frank. 2019. Detecting syntactic change using neural part-of- speech tagger. In Proceedings of the 1st Interna- tional Workshop on Computational Approaches to Historical Language Change, pages 167-174
Andrey Kutuzov, Lidia Pivovarova, and Mario Giu- lianelli. 2021. Grammatical profiling for semantic change detection, In Proceedings of the 25th Confer- ence on Computational Natural Language Learning, pages 423-434.
William Labov. 2011. Principles of linguistic change, volume 3: Cognitive and cultural factors, volume 3. John Wiley& & Sons.
Stefano Montanelli and Francesco Periti. 2023. A sur- vey on contextualised semantic shift detection. arXiv preprint arXiv:2304.00 2304.01666..
Severin Laicher, Sinan Kurtyigit, Dominik Schlechtweg, Jonas Kuhn, and Sabine Schulte im Walde. 2021. Explaining and improving bert performance on lex- ical semantic change detection. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguisties: Student Research Workshop, pages 192-202.
Syrielle Montariol, Matej Martine, and Lidia Pivovarova. 2021. Scalable and interpretable semantic change detection. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguisticy: Human Language Technologies, pages 4642-4651
Mohammad Taher Pilehvar and Jose
Angeliki Lazaridou, Adhi Kuncoro, Elena Gribovskaya, Devang Agrawal, Adam Liska, Tayfun Terzi, Mai Gimenez, Cyprien de Masson d' Autume, Tomas Ko- cisky, Sebastian Ruder, et al. 2021. Mind the gap: Assessing temporal generalization in neural language models. Advances in Neural Information Processing Systems, 14-29348-29363
2019. Wic: the word-in-context dataset for evaluat- ing context-sensitive meaning representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pages 1267-1273
Wenjun Qiu and Yang Xu. 2022. Histbert: A pre-trained language model for diachronic lexical semantic anal- ysis. arXiv preprint arXiv:2202.03612.
Yinhan Liu, Myle Out, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.
Roberta: robustly optimized bert pretraining ap- proach. arXiv preprint arXiv: 1907. 1662.
Xipeng Qiu, Tianxiang Sun, Yige Xu, Yunfan Shao, Ning Dai, and Xuanjing Huang. 2020. Pre-trained models for natural language processing: A survey. Science China Technological Sciences, 63(10):1872- 1897.
Ilya Loshchilov and Frank Hutter. 2017.
Decou- pled weight decay regularization. arXiv preprint
arXiv:1711.05101.
Daniel Loureiro, Francesco Barbieri, Leonardo Neves, Luis Espinosa Anke, and Jose Camacho-C Colladdo.. 2022. Timelms: Diachronic language models from twitter. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics: System Demonstrations, pages 251-260.
Shruti Rijhwani and Daniel Preotiuc-Pietro, 2020. Temporally-inforee inffommed analysis of named entity recog- nition. In Proceedings of the 58th Annual Meeting of the A Association for Computational Linguistics, pages 7605-7617.
Guy D Rosin, Ido Guy, and Kira Radinsky. 2022. Time masking for temporal language models. Proceed- ings of the Fifteenth ACM International Conference on Web Search and Data Mining, pages 841.
Kelvin Luu, Daniel Khashabi, Suchin Gururangan, Kar- ishma Mandyam, and Noah A. Smith. 2022. Time waits for no one! analysis and challenges of tem- poral misalignment. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5944-5958.
Guy D Rosin and Kira Radinsky. 2022. Tempo- ral attention for language models. arXiv preprint 22Xiv:220..02093.

---

Paul Röttger and Janet Pierrehumbert. 2021. Temporal adaptation of bert and performance on downstream document classification: Insights from social media. In Findings the Association for Computational Linguistics: EMNLP 2021, pages 2400-2412.
Maja Rudolph and David Blei. 2018. Dynamic embed- dings for language evolution. In Proceedings of the 2018 world wide web conference, pages 1003-1011
Anastasiia Ryzhova, Daria Ryzhova, and Ilya Sochenkov. Detection of semantic changes in russian nouns with distributional models and grammatical features.
Violeta Seretan. 201 Syntax-based collocation extruc- tion.
Anders Søgaard, Sebastian Ebert, Jasmijn Bastings, and Katja Filippova. 2021. We need to talk about random splits. In Proceedings of the 16th Conference of the European Chapter of the Association for Computa- tional Linguistics: Main Volume, pages 1823- 1923-1832.
Milan Straka and Jana Straková. 2017. Tokenizing,
pos tagging, lemmatizing and parsing 2.0 with udpipe. In Proceedings of the CoNLL 2017 shared universal
lask: Multilingual Parsing from dependencies, pages 88-99.
Zhaochen Su, Zecheng Tang, Xinyan Guan, Juntao Li, Lijun Wu, and Min Zhang, 2022. Improving tem- poral generalization of pre-trained language mod- cls with lexical semantic change. arXiv preprint 2..i::2200.11127
Nina Tahmasebia, Lars Borina, and Adam Jatowtb. 2021. Survey of computational approaches to lexical seman- tie change detection. Computational approaches to semantic change, 6:1.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Aune Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and effi- cient foundation language models. arXiv preprint arXiv:2302.33971.
Adam Tsakalidis, Marya Bazzi, Mihai Cucuringu, Pier- paolo Basile, and Barbara McGillivray, 2019. Min- ing the uk web archive semantic change detection, In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2019), pages 1212-1221.
Jindong Wang, HU Xixu, Wenxin Hou, Chen, Runkai Zheng, Yidong Wang, Linyi Yang, Wei Ye, Haojun Huang, Xiubo Geng, al. 2023. On the robustness of chatgpt: An adversarial and out-of- distribution perspective. In ICLR 2023 Workshop Trustworthy and Reliable Large-Scale Machine Learning Models.

---

A Implementation of Experimental Setup
Evaluation Metric We utilize the Fl score the evaluation metric in all the experiments.
A.1 Task Description
B Implementation of the Baselines
The task of political affiliation classification (POLiAFF) revolves around determining the politi- cal alignment of individuals based on given text. The model is fine-tuned to categorize the content into Republican or Democrat, providing insights into the individual's political leaning. In contrast, named entity recognition (TwINER) focuses on the identification and classification of named en- tities within text. The model is tasked with rec- ognizing and categorizing text into three distinct entity types: people (PER), locations (LOC), and organizations (ORG). Furthermore, political sub- reddit prediction (PSP), derived from the Reddit Time Corpus (RTC) dataset, is a five-way clas- sification task. The goal is to categorize pro- vided text into of five political subreddits: r/donald, r/libertarian, r/conservative, r/polities, and r/chapotraphouse.
This section further provides experimental details of several TLMs used as baselines in Section 4.2: TAda, Temporal, and LMLM. Following the strat- egy by (Su et al., 2022), LMLM utilizes 500 can- didate words and mask ratio of 0.3. For the re- maining baselines, the mask ratio aligns with the standard BERT (Devlin et al., 2019) configuration at 0.15.
TAda The TAda (Röttger and Pierrehumbert, 2021) approach underlines the importance of re- flecting temporal dynamics by training distinct models, each tailored to specific time intervals. To this end, we initiate ensemble of models, each trained unlabeled data corresponding cific year. Each model, or "yearly expert," thus becomes specialized the linguistic nuances of its respective time bucket. During testing, a test input is provided to the model that matches its timestamp. If the timestamp of the test data falls outside the range of the training data, we choose the model trained on the closest year to make predictions. This setup enables the model to adapt to temporal changes by dedicating separate models different time periods.
spe-
A.2 Model Training & Hyper-Parameters
Architecture For all our experiments, utilize the Hugging-Face transformer package"? to imple- ment language models. Our architecture is the BERT-base uncased model, based on a large cor- pus of English data with MLM objective for pre- training. The model includes 12 transformer layers and 12 attention heads; the hidden layer size is 768, and the parameter number is 1 110 million in total. For the downstream classification task, we add linear layer after the last BERT layer. The output is generated using softmax. The maximum input sequence length that the model can handle is 128.
Temporal On the other hand, the Tempo- ral (Dhingra et al., 2022) baseline trained on the entire dataset as a single model. The unique feature of this sctup is the way it incorporates time into the input. Specifically, the model takes in a concatenation of the time and the input x, that is, P(ylær, 1; 0) P(yl t ① x;; 0). This is achieved by prefixing the input with simple string representa- tion of time, such "year: 2014". Thus, the model is trained to generate outputs based on both the input and its associated time, allowing it to develop a sense of temporal sensitivity.
Training details During the pre-training stage, we set the candidate word counts & to 200 and the masking ratio to 30%. We analyze different masking strategies in Appendix 4.4. In both the pre-training and fine-tuning stages, we employ the cross-entropy loss as the objective function and utilize the AdamW (Loshchilov and Hutter, 2017) () optimizer. The learning rate is set to 5e-5, and the weight decay is set to 0.01. To prevent overfitting, we apply a dropout probability of 10% for regular- ization. The model is pre-trained for five epochs and fine-tuned until convergence. We use batch size of 128 and conduct the experiments on eight NVIDIA A5000 GPUs.
LMLM In the lexical-based Masked Language Model (LMLM) (Su ct al., 2022) setup, the model is trained to account for semantic shifts in words over time. This is achieved by constructing a lexi- con of words that have exhibited significant seman- tic changes over time and then using this lexicon in masked language model setup during pre- praiinain
18 "httpps:/sccikii www.llleaannrr.................e wtt.:or///aabeemmmllleeeeee generated/skleamm metrics. f1_score, html
Whttps: //tggiggggggggccccco

---

TWINER
Cos. (1)
2015
2016
2017
2018
SyC.
Methods
2016
2017
2018
2019
Avg.
0.00~~0.1
328
342
251
298
micro
0.01~0002
115
89
139
123
medium
Stanza
81.78
61.85
61.18
68.27
68.27
0.02~~0.03
35
28
44
35
great
UDPipe
78.50
60.59
59.41
60.60
64.78
0.03~1
great
Table 9: Results of different syntactic methods under the time-stratified settings (average of runs).
Table 10: Distribution of the syntactic changed words,
where SyC. represents for the Syntactic Change.
Year
ADJ
ADV
INIJ
NOUN
NUM
PROPN
VERB
The model is thus trained to predict the original word based on the context and the time token, al- lowing it to capture temporal dynamics in word semantics.
1,073
74,074
2016
1,715
0
26,718
12
68,690
6,887
2017
817
0
41,173
0
70,919
4,651
2018
673
493
25,688
0
56,576
4,171
Table II: The distribution of the selected lexicons by Part of Speech (POS) across various years in the WMT pre-trained dataset.
For all these baselines, we perform fine-tuning on the same downstream task data, ensuring fair comparisons among the models. Moreover, the same BERT-BASE-UNCASED pre-trained model the foundational language model for all base- lines, setting level playing field.
D Token-level Analysis
From Figure 5, it is surprising that there is no pos- itive correlation between the word counts and the model's performance. To understand the reason behind this phenomenon, we seleet the top 500 words from the candidates W Waasss according to the cosine value. The distribution of those words is shown in Table 10, which indicates that only about 20% words have relatively significant syn- tactic change (cosine value > 0.01). This suggests that performance mainly comes from correctly pre- dicting small number of keywords, such as topic words and newly emerging words, which have rel- atively salient syntactic change. Furthermore, we also show the distribution of these lexicons by Part of Speech (POS) across various years in Table 11. From the data, it's evident that the distribution of POS remains relatively consistent year-over-year Nouns dominate the distribution, implying their higher propensity for syntactic variation.
As variant of LMLM and SG-TLM, we also consider an additional baseline approach where timestamps are treated as a prefix at the beginning of the sentence and are randomly replaced with the <MASK> token for prediction. Meanwhile, the remaining tokens within the sentence are randomly masked. This baseline is equivalent to employing Temporal-Aware Masking (TAM) with random to- ken masking in sentences, corresponding to the periment with 0 SMLM presented in Section 4.4. Notably, despite these different masking conditions, our proposed method SG-TLM consistently outper- forms the other models.
C Parsing Toolkit Analysis
Since there a strong correlation between the pars- ing toolkit's ' capability and the performance of our SG-TLM, we compare the selected Stanza with another commonly adopted parsing tool, i.e., UD- 19 As shown in Table 9, the results of utilizing Stanza as the parsing method outperforms UDPipe in all the timestamps, i.e., the average accuracy of Stanza is 68.27, while UDPipe is 64.78, which in- dicates that utilizing a more accurate parsing tool can significantly improve the model's performance. Though the UDPipe does not depend on GPU re- sources, this toolkit is unsuitable for parsing syn- tactic roles in the selection process of lexicons.
E Superiority in Adapting Temporal Change
This section aims to demonstrate the superior per- formance of the SG-TLM model in adapting to temporal changes compared to other Temporal Lan- guage Models (TLMs). Specifically, we compare the SG-TLM model against established baselines, i.e., Uniform, Temporal, and LMLM, as previously introduced in Section 4.2. All models are further pre-trained on BERT using 4 million data instances spanning from 2015 to 2018 in the WMT dataset and evaluated to predict masked tokens at differ- ent timestamps using 2,000 data samples from the
19 UDPipe (Straka and Straková, 2017) utilizes fast transition-hanl hased neural dependency parser that follows the same annotation schemes as Stanza.

---

PP2
20-
15
10
5-
2035
2036
2017 2018
2019
2020
Year
Figure 6: Results of adapting to the temporal shift. The SG-TLM consistently achieves the lowest ppl. scores compared Uniform, Temporal, and LMLM models from 2015 to 2020.
same source spanning six years (2015-2020). This experimental setting has also been used verify the ability adapt to temporal changes Röttger and Pierrehumbert (2021). The results this com- parison are presented Figure We can ob- serve that the SG-TLM consistently outperforms the other models across all examined years, achiev- ing the lowest perplexity scores, which illustrates the superior adaptability of the SG-TLM model temporal shifts. Notably, the SG-TLM model also exhibits superior generalization capability with respect to the 2019 and 2020 data that were not present during the training period, outperforming other baseline models, which further demonstrates its robustness and reliability in handling temporal shifts.


Reviews:
[{"review_id": "XylIL1jjYF", "reviewer": "Reviewer_uyuP", "summary": "", "questions": "A:  The techniques seem to imply that the domain of the data/task and as a result the vocabulary remains fixed over time. It is not very clear to me how new words that are added over time would impact the performance of the proposed techniques. Ex. words like Moderna, covid-19 when considering sentiment analysis task related to vaccines over time.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer uyuP,\n\nThank you for your thoughtful review and comments on our paper. We provide the following answers for your concerns.\n\n> Question1: Additional qualitative analysis as to how/why the techniques lead to an improvement would be nice to include in the paper.\n\nAnswer1: We appreciate your point on enriching the paper with a qualitative analysis. \n\nIn response, we've used the running example `”Oval\"` from Figure 1 throughout the paper to clarify how our techniques lead to improvement.\n\nAdditionally, we extend our analysis to explore the `”Oval”` influence on downstream tasks. Take **TWINER** (named entity recognition) for example: \n\nsentence: `Oval Technologies announced their new product launch (2017)`\n\nlabel: `Oval Technologies`--`ORG`\n\nUniform random masking fails to account for the diachronic change of `“Oval”`, leading to inaccurate entity recognition. However, our SG-TLM, informed by syntactic observations, prioritizes learning the semantic representation of `\"Oval\"`. The end result is its correct identification as an `ORG`.\n\nIn next version of our paper, we plan to provide a more comprehensive qualitative analysis to further underline the effectiveness of our approach in handling syntactic shifts.\n\n> Question2: How new words that are added over time would impact the performance of the proposed techniques?\n\nAnswer2: Our primary focus in this paper revolves around the distributional shifts caused by syntactic role change. The use of a fixed vocabulary in both our proposed model and the compared baselines is grounded in this focus. \n\nHowever, we acknowledge the potential impact of new words like \"Moderna\" or \"covid-19”. Even though these words are new, their sentence patterns **will not dramatically change in the next few years**. Our model's primary strength lies in capturing these structural shifts, making it well-suited to understand and generalize the role of new words based on their contextual usage.\n\nSupporting this, our extensive experiments demonstrate the superiority of our approach in terms of **generalization to future datasets** (as seen in Table 1, Table 2, and Figure 6) and **adaptability to linguistic drifts** (evident from Figure 4). \n\nThese findings highlight that even with a fixed vocabulary, our model's emphasis on structural changes equips it to understand and adjust to the impact of new terms. \n\nHope the above replies can answer your questions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* Very well written paper - clear and easy to read.\n* Well thought out and intuitive motivation for proposed techniques to incorporate temporal information into language models.\n* Experiments demonstrate that the techniques are transferable across model architectures which is encouraging.\n* Efficiency of the technique is also a big plus.\n* Strong empirical improvements over considered baselines.", "reasons_to_reject": "* Additional qualitative analysis as to how/why the techniques lead to an improvement would be nice to include in the paper.", "questions_to_authors": "A:  The techniques seem to imply that the domain of the data/task and as a result the vocabulary remains fixed over time. It is not very clear to me how new words that are added over time would impact the performance of the proposed techniques. Ex. words like Moderna, covid-19 when considering sentiment analysis task related to vaccines over time.", "paper_topic_and_main_contributions": "* The paper proposes a novel syntax-guided temporal language model (SG-TLM) with two masking strategies - syntax guided masking and temporal aware masking to incorporate temporal information in pertained language models.\n* It introduces a syntax-guided lexicon selection method to identify words that have undergone significant syntactic changes over time and these words are masked during post-tuning stage.\n* The proposed SG-TLM outperforms other baselines for incorporating temporal information in  language models on downstream tasks. It also demonstrates improved memorization and generalization abilities.\n* Experiments show SG-TLM is more efficient for lexicon selection compared to semantic-based methods, with 5.5x faster parsing and 180x faster measurement.\n* Analysis indicates SG-TLM consistently improves performance when transferred to different model frameworks (encoder-only, decoder-only models).", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ivQoKwLQ3L", "reviewer": "Reviewer_PDRu", "summary": "", "questions": "Consider points 2 and 3 in Reasons to reject - I put in a lot of questions there.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer PDRu,\n\nThank you for your detailed feedback and insights on our paper. Below, we have organized your concerns and our corresponding answers:\n\n> Concern 1: Results and interpretations would benefit from enhanced statistical validation.\n\nAnswer1: **All the results presented in the paper are derived from an average taken over five random seeds**. Here are the results on the **RTC** dataset with the annotated standard deviations (S.D):\n\n| **Model**  | **17-04**  | **18-04**  | **19-08**  | _**20-01**_ | _**20-02**_ |\n| ---------- | ---------- | ---------- | ---------- | ----------- | ----------- |\n| BERT       | 49.61 ±0.8 | 41.74 ±1.1 | 38.23 ±0.5 | 38.84 ±1.2  | 39.97 ±0.3  |\n| + Uniform  | 51.97 ±0.7 | 42.57 ±1.3 | 39.98 ±0.4 | 40.75 ±0.9  | 40.53 ±1.0  |\n| + TAda     | 50.62 ±0.6 | 42.21 ±0.9 | 38.43 ±1.4 | 39.47 ±0.2  | 40.34 ±1.3  |\n| + Temporal | 51.69 ±1.0 | 43.33 ±1.5 | 39.24 ±0.3 | 40.12 ±0.7  | 39.50 ±0.4  |\n| + LMLM     | 51.91 ±0.9 | 43.26 ±1.1 | 39.41 ±1.2 | 40.94 ±0.6  | 40.79 ±0.5  |\n| + STLM†    | 52.48 ±0.6 | 43.67 ±0.2 | 39.60 ±0.8 | 41.21 ±1.1  | 40.52 ±0.9  |\n| + SG-TLM‡  | 52.89 ±0.5 | 43.51 ±0.4 | 41.11 ±0.7 | 41.25 ±0.5  | 40.87 ±0.6  |\n\nIn forthcoming versions of the paper, we will include the S.D across these seeds to further validate the consistency and stability of our model.\n\n> Concern 2: What is the distribution of selected lexicon changes in the pre-training data?\n\nAnswer2: We recognize the importance of a detailed analysis of the selected lexicon and initially placed it in the appendix for readability. However, based on your feedback, we've now included this deeper analysis in the main paper. The table below shows t**he distribution of these lexicons by Part of Speech (POS)** across various years in the WMT pre-trained dataset:\n\n| Year | ADJ   | ADV  | INTJ | NOUN   | NUM  | PROPN  | VERB  |\n| ---- | ----- | ---- | ---- | ------ | ---- | ------ | ----- |\n| 2015 | 1,073 | 19   | 0    | 18,690 | 0    | 74,074 | 3,654 |\n| 2016 | 1,715 | 0    | 0    | 26,718 | 12   | 68,690 | 6,887 |\n| 2017 | 817   | 11   | 0    | 41,173 | 0    | 70,919 | 4,651 |\n| 2018 | 673   | 493  | 1    | 25,688 | 0    | 56,576 | 4,171 |\n\nFrom the data, it's evident that the distribution of POS remains **relatively consistent** year-over-year. **Nouns dominate the distribution**, implying their higher propensity for syntactic variation. We will incorporate more detailed analysis into the main paper in the next version.\n\n> Concern 3: Is this PLM then modeling language change as a general category, or idiosyncratic key word spikes or losses that happen due to news events? \n\nAnswer3: It’s a good question! **New events**, like `the US election`, do cause temporary vocabulary shifts. However, such shifts, though vivid, are typically **short-lived**; when these events are no longer the focal point, language usage will revert to its prior state [1]. Meanwhile, these fluctuations are detectable via straightforward statistical patterns [2]. \n\nHowever, **our study's focus is on the deeper, lasting shifts in language, not just short-term reactions to events.** As illustrated in Figure 2, such changes are challenging to capture purely based on statistical frequency, yet they significantly affect a model's comprehension capability over time. \n\nTo further buttress our claim, we **trace the lexicon evolution** over a span from 2015 to 2020, a duration long enough to transcend the effects of individual events. Within these lexicons, out of all the words selected from each year, **23% showed consistent trends of change**, which validates our primary assumption: our model observing shifts emblematic of broader, long-term linguistic evolution, rather than just transient, event-induced fluctuations.\n\n> Concern 4: Is the method for identifying biases specific to political datasets and not generalizable?\n\nAnswer4:\n\n- **Diversity in Validation Settings:** It's crucial to underscore that our research **isn't solely rooted in political datasets.** In fact, the majority of our experiments, as highlighted in Table 1, 4, 5, and 9, are conducted on the WMT dataset. This dataset offers **a wide array of news sources** that span various topics, domains, and styles, extending beyond just politics. It’s worth noting that our chosen datasets are **standard benchmarks** for studying temporal generalization and are **widely recognized** for evaluating a model's capability in capturing and understanding temporal shifts.\n- **Generalization on unseen data:** SG-TLM also performs well on unseen data. Our performance surpasses the baseline models (uniform vs. SG-TLM): **Poliaff** (2019: 55.75 vs. 57.41, 2020: 46.57 vs. 50.32), **TWINER** (2019: 59.07 vs. 61.87), **RTC** (2020-01: 40.75 vs. 41.25, 2020-02: 40.53 vs. 40.87). **It is worth noting that our model was tested on the brand new data it hasn't seen during training.** If our model was only good at remembering past specific language trends tied to events, it wouldn't have done well on this new data. This can also prove that our model understands broader language changes and don't just memorize past events.\n\n> Concern 5:  capture the fact that certain real-world events lead to some distributional changes in language use, but not a shift or change in the word itself.\n\nAnswer5: As outlined in Answer 3, our study's focus is on the **deeper, lasting shifts** in language, substantiated by our experiments on lexical evolution (as seen in Answer 3) and data generalization (as seen in Answer 4). Therefore, we believe the terms \"language change\" or \"linguistic drift\" are not an overstatement.\n\nTo further address your concerns, the tables below show two main ways `\"oval\"` has been used: as an 'amod' (adjectival modifier) and an 'nmod' (nominal modifier). By comparing **the counts and examples across years**, we can see trends and shifts in usage.\n\n| Syntax Label | 2015 | 2016 | 2017 | 2018 |\n| ------------ | ---- | ---- | ---- | ---- |\n| amod         | 82   | 142  | 264  | 232  |\n| nmod         | 192  | 75   | 60   | 46   |\n\n|      | **amod (adjectival modifier)**                               | **nmod (nominal modifier)**                                 |\n| ---- | ------------------------------------------------------------ | ----------------------------------------------------------- |\n| 2015 | She bought an **oval** table for her dining room.             | The beauty of the gem's **Oval** surpassed her expectations. |\n| 2016 | The artist released a series of paintings on **oval** canvases. | The window's **oval** has an intricate design.              |\n| 2017 | The designer released a watch with an **oval** face.         | Jewelers admire the brilliance of the diamond's **Oval**.   |\n| 2018 | She selected an **oval** frame for her portrait.             | The park's **oval** is used for various events.             |\n\n​\tAnalysis reveals:\n\n- **Shift Towards Simplification**: We find a marked increase in the use of `\"oval”` as `amod`. This is because using `\"oval\"` as an adjective straightforwardly describes an object's shape. **Language's shift towards simplicity** drives the increasing frequency of this usage, e.g. `She selected an oval frame for her portrait`.\n- **Decline in Nominal Modifier Usage**: The declining trend of `\"oval\"` being used as `nmod` might suggest its lesser practical application in daily communication. This can be viewed as a reflection of the natural evolution of language, uninfluenced by specific events.\n\nIn conclusion, the changes in the usage of `\"oval\"` extend beyond specific events. Our qualitative analysis also shows that the pairing with `\"oval office”` only **appeared 3 and 4 times** in 2016 and 2017 respectively even durning the US election. While we acknowledge that significant events can lead to linguistic shifts, our analysis suggests that the trends noted are more likely a reflection of the organic evolution of language. \n\nThe size and font of Figure 1 have been adjusted accordingly.\n\nAgain, thanks for your reviews, and we will definitely put these discussions in the new version. Hope the above answers can help address your concerns.\n\n[1] de Saussure, Ferdinand. *Course in general linguistics*. Columbia University Press, 2011.\n\n[2] Röttger P, Pierrehumbert J. Temporal adaptation of BERT and performance on downstream document classification: Insights from social media, EMNLP-21", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper's aim, and proposed solution are clear and I believe novel. Furthermore, the authors implement the syntax guided temporal masking solution as detailed, and show its advantages over existing solutions. While highly specific towards syntactic biases in data, I can imagine this technique being more generalizable towards countering other biases.", "reasons_to_reject": "1. The results and the interpretations would be bolstered with statistical tests, or at the very least, multiple runs with means and S.D listed. I understand that these are huge models and each fine-tuning experiment might take hours. It is just harder to interpret these numbers as being intrinsically better when they are so close to one another, like in Figure 4.\n2. Table 10 and its related discussion is a little worrying. Firstly, I think some preliminary analysis of the actual selected lexicon should be in the main paper and not the appendix. What is the distribution of selected lexicon changes in the pretraining data? How many nouns, verbs etc. Which are the most common role changes?..There are many questions there of course. Furthermore as Appendix E states, most of the syntactic change is from a few keywords. Is this PLM then modeling language change as a general category, or idiosyncratic key word spikes or losses that happen due to news events? The latter are a subset of the former, and perhaps might be one reason for lower generalizability results (even with the SG-TLM model being the best).\n3. Continuing from my previous point, I am worried that the method for identifying structural biases is not very generalizable, and is dataset specific. For instance, the datasets used in the paper are political datasets. The word 'Oval' I would hazard changes roles leading up to a US election, or in turbulent administrations like Trump (which might explain the biases shown in Figure 1). I feel calling this `language change` or `linguistic drift` is too strong a claim. It implies that people who the dataset represents, genuinely evolved in their usage of that lexicon. But is that actually the case? It rather seems to capture the fact that certain real-world events lead to some distributional changes in language use, but not a shift or change in the word itself. \n\nIt might seem a trivial point, but Table 10 does lead me to think that perhaps using another phrase from 'language change', as well as orienting the pitch of the paper a little differently might help counter this perception that I get.", "questions_to_authors": "Consider points 2 and 3 in Reasons to reject - I put in a lot of questions there.", "paper_topic_and_main_contributions": "The authors target the static nature of current Pre-trained Language Models, most of which are trained on snapshots of language data with no temporal information. Building on existing work which simply adds timestamps, the authors identify that existing PLMs exhibit structural biases present in their training data. Their remedy proposes to use this syntactic bias to direct masking while pretraining, while concurrently adding temporal masking. These lead to noticeable improvements over existing models, both in terms of memorization and generalization.\n\nThe major contributions of this work are:\n- identifying generalizability weaknesses of current PLMs due to syntactic role biases for lexical items in the dataset.\n- proposing a novel method for lexicon selection that chooses a set of lexical items that change substantially in syntactic role over a period of time.\n- Pre-training an MLM with targeted masking of words in their lexicon, as well as temporal masking, and showcasing the improvements of such a PLM on relevant tasks.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "None."}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Figure 1 is tiny and compressed, should be bigger in camera ready. Also, mention the data you extracted the counts from when referring to Figure 1, in caption and perhaps Line 81. The numbers in all the charts need to be set in a larger font as well."}}, {"review_id": "IRilPN5q1x", "reviewer": "Reviewer_shTU", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer shTU,\n\nThank you for the insightful comments and suggestions. We value the input and would like to address your concerns as follows:\n\n**Dependency on Parsing Toolkit**: We acknowledge that the \"Syntax-Guided Lexicon Selection\" module's performance depends on the accuracy of the Parsing Toolkit. However, it's essential to note that modern parsers are highly **effective, efficient and easily accessible**, making them **reliable** for many NLP tasks. We also perform preliminary experiments with different parsers to ensure we use the most accurate tool in Appendix C. \n\n**Balancing Syntactic and Semantic Dimensions in Temporal Information Capturing**: It is a nice suggestion! As you pointed out, there exists scenarios where the semantic information of certain words undergoes a change, while their syntactic roles remain constant.\n\nTo tackle the challenge you've highlighted, we conducted basic experiments, as detailed in the limitations section, to combine syntactic and semantic aspects. Admittedly, these initial attempts did not yield the desired integration.\n\nThis is an unresolved yet compelling issue that requires further research. We will next explore the interaction from multiple dimensions of linguistic attributes and investigate which type of information has  advantages in modeling language evolution.\n\n**Clarity on SMLM in Section 2.2:** \n\nThank you for pointing out the need for clarity in Section 2.2. Here is a detailed explanation:\n\n- **Identification of Unified Set of Keys**: We start with two syntactic dictionaries, $R_1$ and $R_2$, corresponding to different time periods. A unified set of keys, denoted as $S$, is formed by taking the union of keys from both dictionaries: $S = \\text{keys}(R_1) \\cup \\text{keys}(R_2)$.\n- **Frequency-based Refinement**: We then refine $S$ based on a frequency threshold. Only keys with frequencies exceeding 5% of the total number of keys in $S$, resulting in a refined set $S’$: $S' = | k \\in S: \\text{frequency}(k) > 0.05 \\times \\text{total number of keys in } S |$.\n- **Alignment of Dictionaries**: The dictionaries $R_1$ and $R_2$ are adjusted to align with $S’$. For each key in $S’$, if it exists in $R_1$ and $R_2$, its value is retained. Otherwise, it's assigned a value of zero: $R'_1[k] = R_1[k] \\, \\text{if } k \\in R_1 \\, \\text{else } 0; R'_2[k] = R_2[k] \\, \\text{if } k \\in R_2 \\, \\text{else } 0$.\n- **Vector Formation**: Finally, vectors $a_1$ and $a_2$ are formed using the values from the aligned dictionaries $R_1’$ and $R_2’$: $a_1 = \\text{values}(R'_1); a_2 = \\text{values}(R'_2)$.\n\nFinally, we want to thank the reviewers again for their valuable feedback. Hope that the above explanations address your concerns.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.\tThough simple, the idea of capturing temporal information via salient syntactic change distribution is novel to me. The competitive results demonstrate its validity while achieving a higher efficiency.\n2.\tThis paper is well organized and pictures are clear to understand. Thorough experiments are conducted and fully demonstrated the effectiveness of SG-TLM and its each component.\n", "reasons_to_reject": "1.\tSince module “Syntax-Guided Lexicon Selection” needs to parse the syntax for each sentence, it depends heavily on the accuracy of Parsing Toolkit, which can be a major influence on overall F1 score.\n2.\tAs mentioned in section 2.2, author conducts a strong assumption that temporal information can be captured in the changes of syntactic role across time. But it neglects the situation that some words’ semantic information change while their syntactic role remains, thus temporal information cannot reflect accurately from the syntactic role distribution. As mentioned in limitations, both lexical semantic and syntactic information need to be integrated.\n3.\tIn section 2.2, though the SMLM makes sense to me, the way to convert syntactic dictionaries to feature vectors is not clear(What do two vectors a_t exactly indicate?)", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper aims to improve the efficiency of continue training on Temporal PLMs. It launches a thorough study to explore the effects of different methods for lexicon selection and proposes a simple syntax-guided temporal language model to capture time information in syntactically changed tokens. It achieves SOTA performance over baselines with a higher efficiency.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Proposes a continued pretraining scheme that tries to incorporate temporal (date) information into representations in an effort to better generalize on down-stream tasks where the year is given in addition to input text. The method involves using off-the-shelf parsers to guide masking in BERT-like MLM. While performance appears to improve on selected downstream tasks (TwiNER and Poliaff) compared to baselines such as continued pretraining with prepending the date to the input text. 

One concern that was raised was that the work depends on a well-working parser. This happens to work well for well written news, but may suffer for less well-formed text. It is unclear how general this can be beyond well-written text. Furthermore, a more comprehensive evaluation involving more tasks would be more convincing.
Decision: Accept-Findings