Time-Aware Language Modeling for Historical Text Dating
Authors: Han Ren
Hai Wang
Yajie Zhao
Yafeng Ren
Venue: EMNLP 2023 Findings (2023)
Keywords: Text Dating
Diachronic Text Evaluation
Time-Aware Language Model
Temporal Adaption
Hierarchical Model
TL;DR: This paper proposes a language model for temporal word representation learning, and build a hierarchical document model based on it for dating long diachronic documents.

Abstract: Automatic text dating(ATD) is a challenging task since explicit temporal mentions usually do not appear in texts. Existing state-of-the-art approaches learn word representations via language models, whereas most of them ignore diachronic change of words, which may affect the efforts of text modeling. Meanwhile, few of them consider text modeling for long diachronic documents. In this paper, we present a time-aware language model named TALM, to learn temporal word representations by transferring language models of general domains to those of time-specific ones. We also build a hierarchical modeling approach to represent diachronic documents by encoding them with temporal word representations. Experiments on a Chinese diachronic corpus show that our model effectively captures implicit temporal information of words, and outperforms state-of-the-art approaches in historical text dating as well.

Twar- Language Modeling for Historical Text Dating
Han Ren, Hai Wang, Yajie Zhao and Yafeng Ren
Guangdong University of Foreign Studies, China taaneen,,aangaizhaayyiiii @gduffs.ed.... renyafeng @ whu.edu.cn
Abstract
torical documents for the construction of digital librariex(Baledent et al., 2020). Such task also called historical text dating(Boldsen and Wahlberg, 2021), diachronic text evaluation(Popesc) and Strapparava, 2015), or period classification(Tann and Kübler, 2021). Compared to other dating tasks, historical text dating is more challenging as explicit temporal mentions(e.g. time expressions) that help to determine the written date of document usually do not appear it(Toner and Han, 2019). To solve it, current research on historical text dating focuses on document modeling, trying to find the relation- ship between time and linguistic features(Boldsen and Wahlberg, 2021).
Automatic text dating(ATD) a challenging task since explicit temporal mentions usually do not appear in texts. Existing state-of-the- art approaches learn word representations via language models, whereas most of them ig- nore diachronic change of words, which may affect the efforts of text. modeling. Mean- while, few of them consider text modeling for long diachronic documents. In this paper, we present time-aware language model named TALM, to learn temporal word representations by transferring language models of general domains to those of time-specific ones. We also build a hierarchical modeling approach represent diachronic documents by encod- ing them with temporal word representations. Experiments on Chinese diachronic corpus show that model effectively captures im- plicit temporal information of words, and out- performs state-of-the-an approaches in histori- cal text dating as well. Our code is available at: https:/gittbbb.com/caaeriiii/ioo -aateriiii
There are two main issues to historical text mod- eling. One is to learn word representations by di- achronic documents. Current research on word representation either learn static word embedding throughout the corpus(Liebeskind and Liebeskind. 2020; Yu and Huangfu, 2019), or learn dynamic word representations using pre-trained models and Kübbler, 2021). However, neither of them takes into account the relation between time and word meaning. For example, broadcast usually refers to sowing seeds before the 20th century; after that, it means transmitting by radios or TVs in most cases. Studies on language evolution help find the relationship between the same words in differ- ent time periods, since they often discuss them by mapping them into a same semantic space(Ferri et al., 2018), 8). However, how to apply such methods into document modeling for historical text dating is still unexplored.
1 Introduction
The temporal dimension texts is critical to many natural language processing(NLP)) tasks, such as information retrieval (Kanhabua and Nørvåg, 2016), question answering(Shang et al., 2022; Stricker, 2021), text summarization (Cao and Wang, 2022; Martschat and Markert, 2018), event detec- tion(Sprugnoli and Tonelli, 2019), and sentiment analysis(Ren et al., 2016). Timestamps of docu- ments provide essential clues for understanding and reasoning in time-sensitive tasks, whereas they are not always MauilableChhmbeerg 2012). One way to solve this problem is to predict when doc- ument was written according to its content, which is also known as automatic text dating(Dalli, 2006).
Another is document modeling for histori- cal texts. Initial work on neural network- based document modeling employ convolutional neural networks(CNN) works(RNN) (Liebeskind and Liebeskind, 2020; Yu and Huangfu, 2019), while recent research turns to pre-trained models like BERT(Tian and Kübler, 2021) or RoBERTa(Li et al., 2022). However, these
recurrent neural net-
Text dating has been widely introduced in com- putational sociology and digital humanities stud-
ies. One typical application of is date his-
Cooreesponingg author.

---

studies always treat time prediction target, but not a variable in modeling, which does not help to capture the temporal characteristics of diachronic documents. In fact, evidences from the research on language modeling show that time-based mod- els help to capture semantic change of language, and consequently improve the performance of time- sensitive tasks(Agarwal and Nenkova, 2022; Rosin al., 2022; Röttger and Pierrehumbert, 2021). To this end, some studies attempt to incorporate the time factor into language modeling, e.g., learning explicit temporal expressions via language mod- els(Dhingra et al., 2022; Rosin et al., 2022). How- ever, as mentioned above, these methods are not suitable for historical text dating due to the lacking of explicit time expressions in historical texts.
We propose a time-aware language model for historical texts, which uses the temporal adap- tation approach to obtain time-specific vari- ants of words that are temporally consistent with the documents they belong to, thereby improving the ability to model diachronic doc- uments;
We report the superior performances of our model compared to the state-of-the-art mod- els in the historical text dating task, showing the effectiveness of our model in capturing implicit temporal information of words.
2 Related Work
Automatic text dating follows the research roadmap from traditional machine learning to deep learning technologies, like many other NLP tasks. Early studies employ features by manual work to recog- nize temporal expressions within documents(Dalli, 2006; Kanhabua and Nørvåg, 2016; Niculae et al., 2014), which suffer from the problem of gener- alization and coverage rate. Traditional machine learning methods focus on statistical features and learning models, such as Naïve Bayes(Boldsen and Wahlberg, 2021), EVM(Garrci- et al., 2011) and Random Forests(Ciobanu et al., 2013). Recent studies turn to deep learning meth- ods, and the experiments show their superior perfor- mances compared to traditional machine learning Anes(Kulkan) et al., 2018; Liebeskind and Liebe- skind, 2020; Yu and Huangfu, 2019; Ren et al., 2022). Pre-trained models are also leveraged represent texts for the dating task, such as Sentence- BERT(Massidda, 2020; Tian and Kübler, 2021) and RoBERTa(Li et al., 2022). Pre-trained models show state-of-the-art performances on the text dat- ing task; however, few of them consider the time attribute of words.
In this paper, we present time-aware language model named TALM, trying to introduce the time factor into the modeling procedure for historical text dating. Inspired by the work in language evolu- tion, we propose to learn word representations over documents of different time periods separately. In this way, each word has time-specific variants to the vocabulary. It is important because a document should be represented by word embeddings that are temporally consistent with it. We also apply an alignment approach to map all the time-specific variants of a word into the same semantic space in order to make them comparable. In particular, we propose temporal adaptation, a representation learn- ing approach to learn word representations having consistent time periods with documents they be- long to. This approach attempts to learn temporal word representations based on the knowledge of two aspects: time-specific variants of words and their contexts, which depict the temporal attribute words from multiple perspectives. We also build a hierarchical model for long document modeling, where the temporal adaptation is applicd for word representation. We validate our model on large- scale Chinese diachronic corpus and an English diachronic corpus. Experimental results show that model effectively captures implicit temporal information of words, and outperforms state-of-the- art approaches in historical text dating as well. Our contributions can be summarized as follows:
Language evolution studies explore the issue by modeling words from different time periods. Such work can be categorized into three classes: 1)aaar-- ing word embeddings for each time period sepa- rately, then mapping them into the same space via alignment methods(A Alvarez-Melis and Jaakkola, 201 8; Hamilton et al., 2016); 2) learning word em- beddings for time period first, then using them the initial values to train word embedding for other time periods(Di Carlo al., 2019); 3) learning unified word embeddings by introducing temporal variables into the learning procedure(Tang, 2018; Yao et al., 2018). However, static word embed-
We propose temporal adaptation approach that enables the word representation models to capture both temporal and contextualized information by learning the distributed repre- sentations of diachronic documents;

---

ding has the drawback of dealing with polysemous words (Asudani al., 2023), which may not be suit- able for temporal word representation learning. In addition, there are still few studies on the language evolution modeling for historical text dating.
We follow the idea of Hamilton et al. (2016), map- ping the word representations of different time pe- riods into the same semantic space by orthogonal Procrustes algorithm, which aims to make them comparable. Let W E [Rdx be the word vector matrix at time period t. We use the singular value decomposition to solve for matrix Q, so that the cosine similarity between the word vector matri- from different time periods remains unchanged while minimizing the distance in the vector space. The objective for alignment is shown below:
On the other hand, research on the temporal pre- trained models shows that encoding the temporal information in language modeling is beneficial in both upstream and downstream tasks(Agarwal and Nenkova, 2022; Röttger and Pierrchumbert, 2021). Efforts have been made to build models that can learn temporal information, such as incorporating timestamps of texts into the input of the model (Pokrywka and Graliñski, 2022; Pramanick et al., 2021), prefixing the input with temporal represen- tahion(Dhingra al., 2022; Chang al., 2021), learning the masked temporal Information(Rosin et al., 2022: Su et al., 2022). However, there still insufficient discussion of document modeling without explicit temporal mentions.
R' = argmin | --
(2)
where R c Rddd
3.2 Temporal Adaptation
The meaning of words always changes over time. Hence it is necessary to consider the model's abil- ity to adapt different meanings of words over time. This problem is similar to domain adapta- timm(Ramponi and Plank, 2020), where the model should have the ability of transferring from the source semantic space domain to the target one. the historical text dating task, the time period of the document to be predicted is not known in advance, hence the model needs to determine the word rep- resentations of time period that are adaptive the document. This is called temporal adaptation in our model. This subsection shows the temporal adaptation approach. The middle part Figure represents the temporal adaptation module pro- posed in this paper. The input of this module are documents, which consist of segmented blocks. We encode each word with three embeddings: token embedding, position embedding, and block embed- ding. The block embedding of word is utilized to indicate the sequential information of the sentence positioned. The word embedding is defined as follows:
3 The Approach
this section, we explain the mechanism of the time-aware language modeling approach based on temporal adaptation, and how it is applied to the task of historical text dating. Figure I shows the overall architecture of the proposed model, which consists of three main modules: word representa- tion learning and alignment, temporal adaptation, and diachronic document modeling.
3.1 Word Representation Learning and Alignment
Word representation alignment is a pipeline, where word representations are learned firstly in an un- supervised manner, and then are aligned in the same semantic space. Note that each model for learning word representations is trained respec- tively on the documents of every time period. In our method we use BERT(Devlin al., 2018) as the language model to learn. Let Dd be the docu- ment collection of the (th time-period. Let a sen- tence X = (xf,.-- .h), where X E D', is its length, and xf is the ith token in X Let E' (c(, c,),, where ef is the distributed rep- resentation of 2².- We train the model with the masked language modeling task on each period separately, and extract the embedding layers as word representations:
Emmpat
Etoken
Eposition + Estock
The main part of the temporal adaptation module is a Transformer Encoder. hg represents the hid- den space vector for each token obtained from the Transformer Encoder. One of the purposes of this module is to adapt h, to ε which can make the hidden representations fit temporal domain.
On the other hand, contexts also indicate tempo- ral information, which means modeling contextual information helps to determine the time period a (1) text may belongs to. To this end, we design a
E = -ERT(XX

---

Cuee
Las
h
p(x,)
softmax
valee
Aligumeat
Transformer Encoder
A Atention
gjy
Hierarchical
Docum Enonder
BERT
ef
D
h
Word representation leaming and alignment
Temporal adaptation
Diachronic document modeling
The architecture of our model. The left part is the word representation and alignment module, which is to learn and align the temporal word representation in each time period; the middle part is the temporal adaptation module, which aims learn the time-specific variants of words and their contextual information; the right part is the hierarchical modal for diachronic documents, which incorporate the results of temporal adaptation into language modeling.
Figure I:
masked language modeling task to learn the con- textual information of the input texts. Different from other temporal masked language models, do consider the idea of masking temporal in- formation as they did not obviously appear in the text. Alternately, we use fundamental method by masking a portion of words randomly, based on the assumption that the contexts of word also indicate the temporal information since they are in the same time period. Hence we follow the masked language modeling task in BERT training process.
on the block encoding layer lacks the sequential information of the blocks, employ three types of embedding indicate the word order informa- tion and block order information, as mentioned in equation 3. After block transformer encoding, the model obtains the global block-aware repre- sentations, and put it to the next layer for further learning.
In order to make use of the temporal represen- tations of words in diachronic document model- ing, explore the bridging approaches within the Transformer Encoder layer incorporate the tem- poral feature representation during the hierarchical modeling process. Specifically, three bridging ap- proaches are built:
3.3 Diachronic document modeling
The aim of diachronic document modeling is to obtain document-level representation for long historical texts. The novelty of this approach lies the combination the hierarchical document modeling and temporal adaptation. By doing so, temporal features are incorporated into each layer of the transformer encoder, allowing the hierarchi- cal model to learn implicit temporal information according to the knowledge gained from tempo- ral adaptation. Experiments show that this method improves the performance of the text dating task. Inspired by (Grail et al., 2021), we propose hierarchical model for long diachronic document modeling. The hierarchical structure in this paper consists of word transformer and block trans- former. The word transformer is responsible for learning local token representations each block, while the block transformer learns the contextual information of blocks. Since transformer encoder
Feature Aggregation This approach directly adds the input each layer with the temporal representations words.
Dot Attention The document input represen- tation is used the query, while the time feature representations of the words are em- ployed as the key and value. The dot product attention mechanism is applied to each input sample to integrate the temporal information.
A Aditive Attention Similar to dot attention, additional attention parameter matrix is in- troduced in this method. The document input representation serves as the query, while the time feature representations of the words are utilized as the key and value. Using the addi- tive attention mechanism, the temporal feature

---

information weighted and merged into the original input at each step.
number of classes. The cross-entropy computes as:
N Σ
C
1 LELS - N
After N-layer encoding, we utilize an attention mechanism to determine the importance of differ- ent document blocks. Attention scores are calcu- lated to represent the importance of each document block, and these scores are utilized to weigh the feature information of each block. Finally, ob- tain the representation of the document, which is then evaluated for historical text dating.
Mio log Pi.c
(6)
Finally, We aggregate three losses to form the mixture training objectives of our model. The over-
all training objective is formulated as the sum of the above three losses:
L Ccus + LMSE + LMLM
(7)
3.4 Training objective
where LCLS represents the loss function of text doting.both LMSE and LMLM are the loss function of temporal adaptation.
The overall training objective is combination of the losses from modules: temporal adaptation and diachronic document modeling.
Experiments
For temporal adaptation module, we have two training objectives in learning process. One of the learning objective is that transforming hidden rep- resentation learned by Transformer Encoder into target temporal domain, which means minimize the distance between h; and e/. Hence adopt mean squared error our loss function, which is defined below:
This section shows the experimental results as well as discussion of the proposed model. Specifically,
give the details about the experimental settings first, then we introduce the datasets and the prepro- cessing methods for them. In subsection 4.3, we show the results of the comparative experiments. the ablation tests and the parameter evaluations. Finally, a discussion of our model is given.
1 MSE(e,,)) LMSE N
(4)
4.1 Dataset
Our experiments are conducted on two datasets. One is a Chinese dataset including documents of Chinese official dynastic histories(akk.a Twenty- Four Histories) with the time spanning from 2500 B.C. 1600 A.D., and the other is an English dataset named Royal Society Corpus(Kermes al., 2016), with the time spanning from 1660 to 1880. Such kind of datasets have a long time span, so that they are suitable for our model to explore the performance of temporal modeling. Considering that some historical records in the Twenty-Four Histories Corpus were not written in its time pe- riods but compiled and summarized by the later writers, we assign timestamp to each document of the corpus based on the time it is finished. We divided the corpus into smaller blocks for the con- venience of the analysis. Specifically, we extract every thirty sentences from the original corpus one block, with each sentence containing a maxi- mum of thirty Chinese characters or English words. These samples
where N is the number of tokens. By doing so. the model can map the word representations of the input text to the semantic space domain of the corresponding time period, making the model to be represented more appropriate.
Another training objective of temporal adapta- tion module is masked language modeling(MLLM)), which aims to maximize the predict probability of masked words. Let Xn = {M1, ........ T J. which denote the indexes of the masked tokens in the sentence X, where K' is the number of masked tokens. Denote that Xn is the collection of masked tokens in sentence X. and X-n is the collection of unmasked tokens. The learning objective MLM is defined as:
[MLM
logp( Cogk X-
(5)
K
For diachronic document modeling module, we use the cross-entropy loss as the training objective for historical text dating task. In this loss function, Vicc E {0, 1}. representing whether the true class of document i belonging to class c, and C' is the
labeled with their correspond- ing time periods and randomly split into training. validation, and testing sets in an 8:1:1 ratio, respec- tively. Statistical results of the dataset is shown Table 1.

---

Dataset
Parameter Setting During the training stage, the input contains 30 sentences, each of which con- tains 30 words. The dimension of the word vector is 768, and both the temporal adaptation and the hierarchical model has layers, and the dropout rate is 0.1. The batch size of the training is 8 and the learning rate is le-5. The optimizer we use is Year of publication
Blocks
Tokens
Sentences
tions.
Western Han
921
557204
27630
RoBERTa(Li et al., 2022): RoBERTa is an improved version of BERT, which uses larger training corpus and a dynamic mask method to obtain better representation of the text.
Eastern Han
1135
43299]
34050
Western Jin
975
502001
29250
Southern Song
1151
560857
Southem Liang
1871
919178
56130
Twenty- Four
Northern Qi
1826
869476
54780
Histories Corpus
Tang
8098
39782601
Later Jin
3104
1566592
93120
Song
3847
1888350
115410
Yuan
7839
3763214
235180
Ming
2141
1033055
64230
Qing
5242
2433673
1660 1680
2495
741897
74850
1680-1700
2643
755797
79290
1700-1720
304S
7321
1720-1740
2586
676415
775801
1740 1700
4524
1223036
135720
Royal Society
1760-1788
4846
1458099
145380
Corpus
1780 1800
4128
1444720
123840
1800-1820
5590
1684756
167700
AdamW. All the systems conducted over 5-fold cross validation.
1820 1840
6769
2111766
203070
1840-1860
10032
3174480
300960
1860-1850
9061
2870448
Performance Comparison We compare our pro- posed model TALM with other baseline models, and the performance results of each model are shown in Table 2. We use macro precision, macro recall and macro Fl as our metrics. The best- performing values are in bold format.
Table I: Statistical results of the dataset
4.2 Experiment Result
Baselines We conduct experiments on two Histo- ries dataset using our proposed method. All the parameters are randomly initialized and tuned dur- ing the training process. We compare our method with the state-of-the-art models.
As shown in Table 2, on the Twenty-Four Histo- ries Corpus, our proposed model achieves the best performance in the text dating task with an Fl score of 84.99% outperforming other baseline models.
We can see that, the methods based on the struc- ture of pre-trained language models outperform the traditional neural network-based approaches. Longformer achieves a best F1 score of 81.6% in the baseline models, which is specifically used to handle the problem of long text input. However, it is challenging to incorporate temporal information into the Longformer model, as common attention mechanisms have high computation complexity for long document inputs. Hence, in this study, we adopt hierarchical document modeling structure, which reduces the computational complexity for long input sequence. Among the baseline models, both HAN and Hierarchical BERT utilize hierarchi- cal document structures and achieve competitive performance. Our proposed model leverage tempo- ral adaptation and bridging structures in learning process, outperforms these hierarchical models by 6.9% and 3.39% in Fl score respectively, demon- strating the effectiveness of incorporating time in- formation in text dating tasks.
HAN(Yang et al., 2016): Hicrarchical At- tention Network combines word attention and sentence attention mechanisms to obtain document-level representation.
DPCNN(Johnson and Zhang, 2017): Deep pyramid convolutional network is word- level deep convolutional neural network. By using region embedding and residual connec- tion, it can promote network capturing the long-distance dependence between words.
BERT(Devlin et al., 2018): The pre-trained BERT-base model and we finetuned it with the training data for classification.
Longformer(Beltagge et al., 2020): Long- former utilizes self-attention model by us- ing global and sliding window information to model long sequences.
Hierarchical BERT(Khandve al., 2022): Hierarchical BERT uses BERT as the hierar- chical text encoder, and leverage LSTM to encode the sentence feature to obtain the final representation of the document.
In the methods used for dating, SBERT and RoBERTa achieve similar model performance, while LSTM performs a relatively low perfor- mance, due to its limited modeling capability. How- ever, these baseline models primarily focus on the
SentenceBert (Tian and Kübler, 2021): Sen- tenceBERT uses Siamese and triple network architectures to generate sentence representa-

---

language features of the text itself and do not in- corporate the temporal information associated with the text. In contrast, our proposed model incor- porates the corresponding temporal information of the input through different mechanisms. The additive attention achieves the best performance, outperforming SBERT and RoBERTa by 4.28% and 4.24% in Fl score, respectively, demonstrat- ing the advantage of TALM in learning temporal information.
#TM
APC
0.48
4.99
403
Figure 2: The confusion matrix of our model on the Twenty-Four Histories Corpus.
On the Royal Society Corpus, our model achieved the best performance among all the mod- els, similar to that on the Chinese corpus. Specif- ically, our model with additive attention gains an increasing 2.58% F-1 score compared with the best baseline SBERT. It suggests that our model helps to learn temporal information and incorporate this in- formation effectively into text dating tasks in both Chinese and English texts.
We use the metric of Ace, Acc03 and Accã5 to evaluate our model, As can be seen from Table that: 1) our model outperforms the baselines most cases, showing its robustness in both rigid and relaxed evaluation criteria; 2) our model can better distinguish cases of adjacent time periods, compared with the other baselines.
Comparison Of Bridging Methods We use differ- ent bridging approaches to make the model having the ability of time-awareness. Here we will dis- cuss the impact of these approaches to model per- formances. As shown in Table 2, each of these approaches leads to improving performance, showing the effectiveness of integrating temporal information through bridging mechanisms. Over- all, the additive attention approach achieves the best results. This attention mechanism introduces additional attention matrix parameters, which con- trol the amount of temporal word representation information selected during training. As result, the model can learn word vector representations corresponding to specific time periods more effec- tively.
Ablation Study In the ablation experiments, we analyze the contributions of these components by evaluating TALM without time-specific variants of words, TALM without context learning, and TALM without temporal attention.
When the temporal attention module is removed, the performance of the bridging module is also climinated. Therefore, temporal adaptation and time-aware language modeling cannot be inte- grated with the dating task. This model performs the worst with an F1 score of 76.69%. When the -iee-fpecific variants of words are removed, the model does not work well to learn temporal word representations, leading to performance decrease of 3.42% in Fl score. Similarly, when the con- text learning module is removed. the model does not work well learn contextual information for temporal word representations, resulting in per- formance decrease of 6.31 in Fl score. These results indicate that all the module play crucial roles in the text dating task.
Flexible Evaluation Criteria This study focuses on the historical text dating task. However, in the historical corpora, word meaning changes in grad- ual way, and the semantic similarity between texts from adjacent periods may be higher than those that have longer time span. Inspired by (Li et al., 2022), define more flexible evaluation metric called A. This metric treats the prediction result adjacent the correct time period as cor- rect one, which is more flexible metric. Nace@ represents the number of the correct cases, where those having a period class distance of +| _ are also taken into account. Nau represents the total number of samples in the dataset. The specific mathematical formula is defined as follows:
4.3 Analysis
Case Study In this section, we discuss the per- formances of our model in different time periods, analyze cases that our model fail to recognize, and investigate the impact of the temporal adaptation module in visualization way. Figure shows the confusion matrix of our model on the test set. We can see that it is much difficult to predict the time period of the Southern Liang. Actually, most samples are falsely predicted to the Dynasty Tang,
Noce @K
A
Nall

---

Twenty-Four Histories Corpus
Royal Society Corpus
Method
F1
Acc
Acc@3
Ace
R
Ace
Acc@3
Acc@5
Yani et al., 2016)
77.09
79.12
78.09
81.32
92.23
96.88
56,84
56.78
56.45
59.03
88.17
94.65
DPCNNUJohsson Zhang. 2017)
75.57
90.63
96.38
53.64
56.02
BERT(Devlin al. 2018)
81.11
80.07
80.59
83.72
92.29
97.36
57.63
57.27
57.27
59.27
87.71
95.01
SSGC(Zuu and Koniusz, 2020)
80,76
81.44
81.10
82.20
90.05
97.15
Longformen( Bellugy et al., 2020)
81.83
81.37
81.60
82.92
90.34
95.84
57.93
58.21
57.41
59.83
89.00
95.24
Hierarchical BERT (Khandve et al., 2022)
81.59
79.47
80.16
82.34
91.12
95.73
45,52
45.66
45.41
47.45
75.90
86.01
LSTM(YU and Huangfu, 2019)
68.62
68.99
89.25
95.
53.99
53.61
55.62
88.81
95.32
SBERTTTian and Kübler, 2021)
80,50
80,92
80,71
81,80
91.20
96.72
57.92
59.17
58.19
60.09
88.00
94,63
RoBERTa(Li et al., 2022)
80.42
81,08
80.75
82.23
91.14
97.31
57.92
58.05
57,52
59.71
88.87
94.94
Teature aggregation
80.63
81.68
81.15
83.64
91.33
96.50
58.11
58.04
57.86
59.64
87.57
94.16
attention
82.19
81.87
82.03
83.96
92.02
96.82
56.72
56.74
56.56
58.86
87.01
94.25
additive
93.50
61.38
62.81
Table 2: Performances of our model and baselines the Twenty- -our Histories corpus and Royal Society Corpus.
The evaluation metrics include P, R, Fl, Acc, Acc@3 and Acc@5.
Twenty-Fou History Corpus
Royal Society Corpus
Model
P
R
F1
P
R
F1
TOur method)
84.96
85.02
84.99
61.11
61.38
60.77
w/o time-specific variants of words
81.60
81.54
81.57
58.44
58.39
58.11
w/o context learning
78.73
78.63
78.68
53.74
53.66
53.49
w/o temporal attention
79.62
73.97
76.69
51.32
51.28
50.82
Table 3: Results of ablation study on the Twenty-Four Histories corpus.
TALM
RoBERTa(Lie et al., 2022)
Period
F1
P
Western Han
87. 66
88.03
87.85
93.80
90.60
92.17
Eastern
77.07
82.54
80.34
91.08
Western 90.38
Southern Song
89.61
83.13
86.25
89.86
74.70
81.58
Southern Liang
80.00
59.26
68.09
68.66 68
57.87
62.81
Notthern Qi
69.06
80.30
74.26
54.27
81.81
65.26
Tang
83.48
88.96
86.13
83.70
83.19
83.45
92.70
87.44
Song
81.48
84.54
82.98
74.23
80.22
77.11
Yuan 89.86
95.38
92.54
85.63
92.12
88.76
Figure The document distribution representation vi-
Ming 89.40
88.92
89. 16
68.51
87.03
76.67
sualization of the two models.
Qing
96.70
96.25
96.47
96.52
93.62
95.05
Table 4: the result macro P, R, and Fi-score value each period.
thee--t-thh-rrtt model RoBERTa in those cases who have short historical time duration. Specifically, our model gains an increasing F1 score of 5.28% 28, 9.00% and 13.40% in the time period of Southern Liang, Northern Dynasty of Qi, Later Jin. It also can be seen from Table that, the score of the metric Accō5 of our model significantly higher than that of Acc. It suggests that texts of adjacent periods have a great impact on the per- formance of the model, leading to higher prob- ability of the error- prrorcpredicted period results close to the gold labels than to other periods. To make an in-depth analysis of such errors, we sample the false- preeicpredeted cases of Southern Liang, which got the lowest recall score on the test set. As shown in Table 5, the sentence 1-3 belong to the South-
with a few samples predicted as the Northern Qi. This could be attributed to the relatively short time duration of each period in the Southern and North- ern Dynasties. In other words, the semantic gap between these three dynasty is not much obvious. Furthermore, the Later Jin period lasted for only about ten years in the ancient China history, and this may lead to low performance in identifying if a document belongs to this time period. This indicates that the Later Jin period serves as a transi- tional time period with semantic and textual writing styles similar to the Tang and Song Dynasties.
Table shows that our model outperforms the

---

Examgle
petepetttion
Period Interval
K 20EM2.. 2
13.22 AL di
Therefust, milltar''s decision ocsse the offensive and refinia from
Nonthem Qi
Southem Lizng
11
does contrdict pronises predecessos. occupyang 1.1.
W8... 2AII 5
Success om only be achieved through homesty, while being confosel
1032 4888 U. is 8 77000-5---
Northen Qi
Southem Linng
1
workly matters will the bezioning of disadvaniage
SE ft 2. S2R consecative nights, Noateang was pocibed.
Southem ling
ITT RP. H9 St. 885 35:1011
Snddens from Grass Thinue, and Forr Schools need to wear thosr gammens when vieting
Soug
Late: Jin
+1
JEM FM 175 S05.55550000000 BCR/ SCE.
11-001 6.2-211..
ted aleaaly i 16th 61 To maitraia wealh and sobiiiy to easure the security of the homeland.
City.
Later
WA A
Tang
Taker
Table 5: Examples of errors.
ern Liang period, whereas our model misclassifies them into the class of Northern Qi or Tang, which are adjacent periods to the gold label. Likewise, the sentence 4 and 5 are both texts from the Later Jin period, however the model misjudges them as the texts of the Song period. As matter of fact, there is very short time interval between the Later Jin and the Song period, nearly 13 years. The main rea- son of these errors may lie in the high similarity of language usage as such periods are very close, and language changes without striking difference are insufficient for more precise prediction. In other words, there is not significant change of words in these adjacent periods, making it difficult to pro- vide discriminative features for temporal language modeling.
cal text dating by building TALM, a time-aware language model. To address the problem of the model suffering from missing temporal expressions in historical texts, we propose a temporal adapta- tion model that makes the model have the ability of time-awareness. We also propose hierarchi- cal modeling method incorporate the temporal adaptation and long document modeling. We con- duct experiments using different attention mech- anisms, showing the effectiveness of integrating temporal information on historical text dating task. Our study provides new evidences to support the viewpoint that encoding temporal information con- tributes to improving the performances of language modeling.
Time-aware Visualization In this section, we further investigate the impact of the temporal se- mantic learning module on the text dating task and compare it with the RoBERTa model. To avoid the influence of the contexts to word representations during the hierarchical document modeling pro- cess, we select the representations obtained after the first layer of additive attention in the document encoding module. After encoding with this layer of attention, the model's 's input incorporates the word representations with temporal information. We take the average of the word representations as the doc- ument vector and used the T-SNE method to visu- alize the document vectors in the test set. Similarly, we also select the word vector input layer of the RoBERTa model for comparison. Figure 3 shows the visualized results, indicating that model has a better performance to evaluate the temporally semantic relationship among documents.
Limitations
In this study, we make our experiments on datasets of two languages, Chinese and English. In or- der to further validate the effectiveness of the pro- posed model, datasets of more language versions are expected in order to investigate the model per- formance on the modeling of multiple languages. Additionally, regarding the division and definition of historical periods, this study adopted a coarse- grained labeling standard. Due to the chronological order of the corpus, coarsegrained labeling may not accurately represent the exact time of the texts. Therefore, in the future, plan to collect struct fine-gra aine-grineed textual corpora improve the performance of temporal information learning and enhance the accuracy of the text dating task.
Acknowledgements
This work supported by Research Fund of Na- tional Language Commission(grant No. 145- 2), National Natural Science Foundation of China(grant No. 61977032).
5 Conclusion
this paper, we propose to incorporate the time attribute into the language modeling for histori-

---

References
Bhuwan Dhingra, Jeremy Cole, Julian Martin Eisen- schlos, Daniel Gillick, Jacob Eisenstein, and William W Cohen. 2022. Time-aware language models temporal knowledge bases. Transactions the Asso- ciation for Computational Linguistics, 10:257-273.
Oshin Agarwal and Ani Nenkova. 2022. Temporal ef- fects on pre-trainet models for language processing tasks. Transactions of the Association for Compula- tional Linguisties. 10:904-921.
Valerio Di Carlo, Federico Bianchi, and Mattco Pal- monari. 2019. Training temporal word embeddings with a compass. In Proceedings of the AAAI con- ference on artificial intelligence, volume 33, pages 6326 6334.
David Alvarez-Melis and Tommi S Jaakkola. 2018. Gromov-wassersee alignment of word embedding spaces. arXiv preprint arXiv:1809.00013.
Deepak Suresh Asudani, Naresh Kumar Nagwani, and Pradeep Singh. 2023. Impact of word embedding models on text analyties in deep learning environ- ment: review. Artificial Intelligence Review, pages 1-81.
Fernando Ferri, Arianna D'Ulizia, and Patrizia Grifoni. 2018. Computational models of language evolution: Challenges and future perspectives. J. Univers. Com- put. Sci., 24(10):1345-15
Anaëlle Baledent, Nicolas Hiebel, and Gaäl Lejeune. 2020. Dating ancient texts: an approach for noisy french documents. In Language Resources and Eval- uation Conference (LREC) 2020.
Anne Farcia-Fernandez Anne-Laure Ligozat, Marco Dinarelli, and Delphine Berhhard. 201 1. When it written? automatically determining publication dates. In String Processing and Information Retrieval: 18th International Symposium, SPIRE 2011, Pisa, Italy October 17-21, 2011. Proceedings 18, pages 221- 236. Springer.
Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020. Longformer: The docummetnnnt transformer. arXiv preprint arXiv:2004.05/1 1550
Quentin Grail, Julien Perez, and Eric Gaussier. 2021. Globalizing bert-based transformer architectures for long document summarization. In Proceedings of the 16th conference of the European chapter of the Asso- ciation for Computational Linguistics: Main volume, pages 1792-1810.
Sidsel Boldsen and Fredrik Wahlberg. 2021. Survey and reproduction of computational approaches to dating of historical texts. In Nordic Conference on Com- putational Linguistics (NoDaLiDa), pages 145-156. Linköping University Electronic Press, Sweden.
Shuyang Cao and Lu Wang. 2022. Time-aware prompting for text generation. arXiv:2211.02162.
William L Hamilton, Jure Leskovec, and Dan Juraf- sky. 2016. Diachronic word embeddings reveal sta- tistical laws of semantic change. arXiv preprint arXiv:
arXiv preprint
Nathanael Chambers. 2012. Labeling documents with timestamps: Learning from their time expressions. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), pages 98-106.
Rie Johnson and Tong Zhang. 2017. Deep pyramid convolutional neural networks for text categorization. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguisties (Volume I: Long Papers), pages 562-570.
Ernie Chang, Yow-Ting Shiue, Hui-Syuan Yeh, and Vera Demberg. 2021. Time-aware ancient chi- nese text translation and inference. arXiv preprint arXiv:2107.03179,
Nattiya Kanhabua and Kjetil Nørväg. 2016. Improving temporal language models for determining time of non-timestamped documents. In International con- ference on theory and practice of digital libraries, pages 358 330- Springer.
Alina Maria Ciobanu, Anca Dinu, Liviu P Dinu, Vlad Niculae, and Octavia-Maria Sulea. 2013. Temporal classification for historical romanian texts. In Pro- ceedings of the 7th workshop on language technology for cultural heritage, social sciences, and humanities, pages 102-106.
Hannah Kermes, Stefania Degaetano-Ortlie, Ashraf Khamis, Jörg Knappen, and Elke Teich. 2016. The royal society corpus: From uncharted data to corpus. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'I6) pages 1928-1931, Portorož, Slovenia. European Lan- guage Resources Association (ELRA).
Angelo Dalli. 2006. Temporal classification of text and automatic document dating. In Proceedings of the Human Language Technology Conference of the NAACL, Companion Volume: Short Papers, pages 29-32.
Snehal Ishwar Khandve, Vedangi Kishor Wagh, Apurva Dinesh Wani, Isha Mandar Joshi, and Raviraj Bhumi- nand Joshi. 2022. Hierarchical neural network ap- proaches for long document classification. In 2022 14th International Conference on Machine Learning and Computing (ICMLC), pages 115-119.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understand- ing. arXiv preprint

---

Vivek Kulkarni, Yingtao Tian, Parth Dandiwala, and Steven Skiena. 2018. Simple neologism based do- main independent models to predict year of author- ship. In Proceedings the 27th International Confer- ence on Computational Linguistics, pages 202-212.
Guy Rosin, Ido Guy, and Kira Radinsky. 2022. Time masking for temporal language models. In Proceed- ings of the Fifteenth ACM International Conference on Web Search and Data Mining, pages 833-841.
Paul Rötter and Janet B Pierrchumbert. 2021. Tem- poral adaptation of bert and performance on down- stream document classification: Insights from social media. arXiv preprint urXiv:210.008116.
Meiwei Li, Yunhui Qin, and Wei Huangfu. 2022. Roberta: An efficient dating method of ancient chi- nese texts. In Workshop on Chinese Lexical Seman- tics, pages 293-301. Springer.
Chao Shang, Guangtao Wang. Peng Qi. and Jing Huang. 2022. Improving time sensitivity for question answer- ing over temporal knowledge graphs. arXiv preprint M.Xi::22030002.
Chaya Liebeskind and Shmuel Liebeskind. 2020. Deep learning for period classification of historical hebrew texts. Journal of Data Mining & Digital Humanities, 2020.
Rachele Sprugnoli and Sara Tonelli. 2019. Novel event detection and classification for historical texts. Com- putational Linguistics, 45(2):229-2 265.
Sebastian Martschat and Katja Markert. 2018. A tempo- rally sensitive submodularity framework for timeline summarization. arXiv preprint vrXivv1810.07940
Armand Stricker. 2021. Question answering natural language: the special case of temporal expressions. In Proceedings of the Student Research Workshop Associated with RANLP 2021, pages 184-192.
Riccardo Massidda. 2020. rmassidda@ @ dadoeval: Doc- ument dating using sentence embeddings evalita 2020. EVALITA Evaluation of NLP and Speech Tools for
17th, 2020, page 403.
Zhaochen Su, Zecheng Tang, Xinyan Guan, Juntao Li, Lijun Wu, and Min Zhang. 2022. Improving tem- poral generalization of pre-trained language mod- els with lexical semantic change. arXiv preprint arXiv:2210. 171.27
Vlad Niculae, Marcos Zampieri, Liviu Dinu, and Alina Maria Ciobanu. 2014. Temporal text rank- ing and automatic dating of texts. In Proceedings of the 14th Conference of the Europeun Chapter of the Association for Computational Linguistics, volume 2: Short Papers, pages 17-21.
Xuri Tang. 2018. A ofate-of-the-ar of semantic change computation. Natural Language Engineering, 24(5):649-65 676.
Jakub Pokrywka and Filip Gralinski. 2022. Temporal language modeling for short text document classifi- cation with transformers. In 2022 17th Conference on Computer Science and Intelligence Systems (Fed- CSIS), pages 121-128. IEEE.
Zuoyu Tian and Sandra Kübler. 2021. Period classifi- cation in chinese historical texts. In Proceedings of the 5th Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature, pages 168-177.
Octavian Popescu and Carlo Strapparava. 2015. Se- meval 2015, task 7: Diachronic text evaluation. In Proceedings of the 9th International Workshop on Se- mantic Evaluation (SemEval 2015), pages 870-878.
Gregory Toner and Xiwu Han. 2019. Language and chronology: text dating by machine learning, volume 84. Brill.
Zichao Yang, Diyi Yang. Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. 2016. Hierarchical at- tention networks for document classification. In Pro- ceedings of the 2016 conference of the North Ameri- can chapter of the association for computational lin- guistics: human language technologies, pages 1480- 1489.
Aniket Pramanick, Tilman Beck, Kevin Stowe, and Iryna Gurevych. 2021. The challenges of tempo- ral aligmment on twitter during crises. arXiv preprint 11X.v22104.08335
Alan Ramponi and Barbara Plank. 2020. Neural unsu- pervised domain adaptation in nlp-a a survey, arXiv preprint 0tt::2006.00333
Zijun Yao, Yifan Sun, Weicong Ding, Nikhil Rao, and Hui Xiong. 2018 Dynamic word embeddings for evolving semantic discovery. In Proceedings of the eleventh aem international conference on web search and data mining, pages 673-681.
Yafeng Ren, Mengxiang Yan, and Donghong Ji. 2022. hierarchical neural network model with user and product attention for deceptive reviews detection. Inf.
Sci., 604:1-10.
Xuejin Yu and Wei Huangfu. 2019. A machine learning model for the dating of ancient chinese texts. In 2019 International Conference on Asian Language Processing (IALP), pages 115- IEFE.
Yafeng Ren, Yue Zhang, Meishan Zhang, and Donghong Ji. 2016. Improving twitter sentiment classification using topic-enriched multi-prototype word embeddings. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, February 12-17, 2016, Phoenix, Arizona, USA, pages 3038 3044. AAAI Press.
Hao Zhu and Piotr Koniusz. 2020. Simple spectral graph convolution. In International conference on learning representations.


Reviews:
[{"review_id": "ybjlW1WAsY", "reviewer": "Reviewer_WaJ5", "summary": "", "questions": "- The max precision of the proposed method is 84.96. What are the cases of False positives?\n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Q1: The authors could provide more details about the dataset that they used to train their model.\n\nA1: Thanks for your comment. Full details of the dataset are not given in our manuscript due to the page limitations, and will be added in the revised version.\n\nQ2: The authors could also provide more details in the analysis sections, especially by showing the examples on where the proposed model fails.\n\nA2: Thanks for your comment. In the revised version of our manuscript, we will give more details about the error cases.\n\nQ3: The authors could explore the use meta temporal features, such as the author of the text and the genre of the text.\n\nA3: Thanks for your comment. There is no doubt that meta temporal features help to text dating, which has been discussed in the related work section. Such features, however, are not taken into account in this work, mainly because we hope that this work can be applied in some real application scenarios, such as dating a historical document without any meta information(it usually happens to ancient China manuscripts in archaeological findings). We are also happy to discuss the contribution of the meta information of a historical document when it is available.\n\nQ4: The max precision of the proposed method is 84.96. What are the cases of False positives?\n\nA4: In most cases, predicted results of false positives are temporally close to the ground truth, according to the confusion matrix in Figure 1. It matches the linguistic intuition that languages of adjacent time periods are more similar than those having a long temporal distance, which also brings the challenge of determining whether a historical text belongs to a time period or an adjacent one. We will show more details of the false positives in the revised manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This is a technically sound paper with detailed evaluation and experimental clarifications. Below are the detailed descriptions:\n\nTechnical soundness: The paper is technically sound. The authors provide a clear and well-motivated explanation of their approach. The experimental evaluation is rigorous and the results are convincing.\n\nEvaluation: The paper evaluates their approach on a large dataset of historical texts. The results show that their approach outperforms existing state-of-the-art models. The authors also provide a detailed analysis of the results, which helps to understand the strengths and weaknesses of their approach.\n\nClarity: The paper is well-written and easy to understand. The authors provide clear and concise explanations of the concepts and methods. The experimental results are also presented in a clear and informative way.\nSignificance: The paper makes a significant contribution to the field of natural language processing. The proposed approach has the potential to be used in a variety of applications, such as historical research and digital humanities.\n", "reasons_to_reject": "1) The authors could provide more details about the dataset that they used to train their model.\n2) The authors could also provide more details in the analysis sections, especially by showing the examples on where the proposed model fails.\n3) The authors could explore the use meta temporal features, such as the author of the text and the genre of the text.\n", "questions_to_authors": "- The max precision of the proposed method is 84.96. What are the cases of False positives?\n", "paper_topic_and_main_contributions": "Summary: The paper proposes a time-aware language modeling approach for historical text dating. The authors argue that traditional language models are not well-suited for this task, as they do not take into account the temporal context of text. To address this, the authors propose a model that jointly models text and its timestamp. This allows the model to learn the temporal patterns of language, which can be used to improve the accuracy of text dating.\n\nNovelty: The paper makes several novel contributions. First, the authors propose a novel time-aware language modeling approach for historical text dating. Second, the authors evaluate their approach on a large dataset of historical texts and show that it outperforms existing state-of-the-art models. Third, the authors provide a detailed analysis of the results, which helps to understand the strengths and weaknesses of their approach.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "NA", "Typos_Grammar_Style_and_Presentation_Improvements": "NA"}, "extra_text": {}}, {"review_id": "QXcVlhPSzF", "reviewer": "Reviewer_izi8", "summary": "", "questions": "It would be beneficial to mention in detail about the weight setting and parameters in more detail. It wasn't available in the paper so it's confusing", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Q1: Ethical and Societal Implications: Although the paper does not discuss potential ethical or societal implications explicitly, the use of language models for historical text analysis could raise concerns about potential biases, misinterpretations, or controversial narratives. The paper should address these considerations to ensure responsible and unbiased historical analysis.\n\nA1: Thanks for your comment. We will consider the improvement to alleviate ethical and societal Implications to achieve an unbiased historical text analysis model.\n\nQ2: Limited Generalizability: The paper focuses on a specific historical context, the Chinese diachronic corpus. The generalizability of the proposed TALM model to other languages, cultures, or historical periods might be limited. This could potentially restrict the applicability and impact of the proposed approach beyond the Chinese historical context.\n\nA2: Thanks for your comments. The aim of this work is to explore the general method for historical context modeling. To this end, neither the research hypothesis on languages, cultures or historical periods is introduced, nor task-specific features are employed in our model. We think there are no explicit clues to show that there is the generalizability problem of our model. Despite that, we are willing to validate it by adding experiments of other datasets and report it in the revised version of our manuscript.\n\nQ3: It would be beneficial to mention in detail about the weight setting and parameters in more detail. It wasn't available in the paper so it's confusing.\n\nA3: Thanks for your comment. Details of weights and parameters are not given in the manuscript due to the page limitations. We will add them in the revised one.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Strengths of the paper:\n\n1. Empirical Validation: The paper's empirical evaluation on a Chinese diachronic corpus demonstrates the practical effectiveness of TALM. Its outperformance compared to existing state-of-the-art approaches in historical text dating underscores the practical utility of the proposed method, making it a valuable addition to the NLP toolkit.\n\n2. Insightful Interpretation of Results: The authors provide insightful interpretations of the observed performance trends. For example, the explanation of the difficulty in predicting the Southern Dynasty Liang due to semantic similarities with other periods and the impact of the short duration of the Later Jin period adds valuable context to the results.\n\nThe discussion of the confusion matrix for the model's predictions, particularly focusing on the challenges of predicting the time period of the Southern Dynasty Liang, demonstrates a thorough analysis of the model's strengths and limitations. This kind of analysis provides a deeper understanding of where and why the model may struggle.", "reasons_to_reject": "1. Ethical and Societal Implications: Although the paper does not discuss potential ethical or societal implications explicitly, the use of language models for historical text analysis could raise concerns about potential biases, misinterpretations, or controversial narratives. The paper should address these considerations to ensure responsible and unbiased historical analysis.", "questions_to_authors": "It would be beneficial to mention in detail about the weight setting and parameters in more detail. It wasn't available in the paper so it's confusing", "paper_topic_and_main_contributions": "This paper addresses the challenging task of automatically determining the temporal context or date of historical texts, especially when explicit temporal mentions are not readily available.\n\nContributions of the paper:\n\n1. Time-Aware Language Model (TALM): The paper introduces TALM, a specialized language model designed to capture temporal changes in word usage. Unlike traditional language models that often overlook diachronic variations, TALM leverages pre-trained models from general domains and adapts them to time-specific contexts. This enables the model to better understand and represent the evolution of language over time.\n\n2. Hierarchical Document Modeling: The authors propose a hierarchical approach to model long diachronic documents. By encoding documents with temporal word representations obtained from TALM, the model captures implicit temporal information present in the text. This hierarchical representation facilitates the understanding of how language and topics change over extended periods.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "modal -> module [285 line]"}, "extra_text": {}}, {"review_id": "7LkSGW8Cvq", "reviewer": "Reviewer_KmTN", "summary": "", "questions": "I have several questions concerning both the method and experiment parts, hoping the author can clarify or make revisions in subsequent versions to enhance the clarity for future readers:\n\n1. What is Eblock in Eq(1) and how it is represented?\n\n2. In Figure (1), how is the learned h_i is used in document modeling? How are the q, k, v built? What is b_k?  By saying \"temporal features are incorporated into each layer\" in Section 3.3, do you mean h_i?\n\n3. In ablation study, by \"w/o\" temporal attention, do you mean remove L_CLS in Eq(6)? If that is the case, how do you generate the prediction y with removing the classification output? If not, can you be more specific what does this w/o mean?\n\n4. In Parameter setting, it's mentioned that training contains 30 sentences while this is not aligned with the dataset showed in Table 1. What are the training sentences here?\n\n5. How is the weights used in 3.1 initialized? \n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Q1: The study, along with the employed baseline comparisons, centers exclusively around Chinese corpus, thereby confining the applicability of the proposed method.\n\nA1: Thanks for your comment. We will validate the applicability of the proposed method on more historical text datasets and report it in the revised manuscript.\n\nQ2: The absence of a comparison with Large Language Models such as the GPT models or Chinese Language Models might be a concern.\n\nA2: Thanks for your comment. We will report supplement experiments of large language models in our revised manuscript.\n\nQ3: What is Eblock in Eq(1) and how it is represented?\n\nA3: Eblock denotes the position of the sentence in a block. It indicates the sentence order.\n\nQ4: In Figure (1), how is the learned h_i is used in document modeling? How are the q, k, v built? What is b_k? By saying \"temporal features are incorporated into each layer\" in Section 3.3, do you mean h_i?\n\nA4: \n1) \"how is the learned h_i used in document modeling?\"\nAttention models are utilized to incorporate temporal features(h_i) into word representations, and the aim is to enable the time-aware ability of document modeling by using word representations with temporal information.\n2) \"How are the q, k, v built?\"\nq denotes the word representation by hierarchical document encoding; k, v denotes h_i. The attention model is to incorporate temporal information into hierarchical document representation.\n3) \"What is b_k?\"\nb_k denotes the kth block of the dataset. In hierarchical modeling, long documents are usually partitioned into multiple blocks. In this paper, each block includes 30 sentences.\n4) \"By saying \"temporal features are incorporated into each layer\" in Section 3.3, do you mean h_i?\"\nYes. We incorporate the weighted h_i from the attention model into each layer of the document encoder.\n\nQ5: In ablation study, by \"w/o\" temporal attention, do you mean remove L_CLS in Eq(6)? If that is the case, how do you generate the prediction y with removing the classification output? If not, can you be more specific what does this w/o mean?\n\nA5: w/o temporal attention means that the attention module in the diachronic document modeling stage in Figure 1 is removed. In other words, it only remains a hierarchical document encoder in the stage. The aim of this ablation test is to explore the effectiveness of the temporal adaptation model.\n\nQ6: In Parameter setting, it's mentioned that training contains 30 sentences while this is not aligned with the dataset showed in Table 1. What are the training sentences here?\n\nA6: Sentences in Table 1 refer to the total amount of sentences in the document of the same time period. While in the training stage, each sample(block) contains 30 sentences.\n\nQ7: How is the weights used in 3.1 initialized?\n\nA7: The weights used in section 3.1 are initialized randomly.\n\nQ8: I understand the limitation of this work is focusing on Chinese historical text dating. It would be beneficial to expand the conversation to encompass potential applications in other languages and provide insights based on the distinct characteristics of each language.\n\nA8: Thanks for your comment. We will continuously consider more application scenarios of our model. We believe that it helps to model more time-sensitive tasks, but not only in the historical text dating task.\n\nQ9: Readers will be interested in the public LLM-based performance which may also be considered as baselines.\n\nA9: Thanks for your comment. We will report supplement experiments of large language models in our revised manuscript.\n\nQ10: For the representation learned in Section 3.1, by splitting the data into different collections, it's making the data more sparse. I wonder if it is a better way to pretrain the model on the entire dataset, followed by a subsequent training within each collection.\n\nA10: Thanks for your comment. The idea of splitting the data into different periods is inspired by the research on lexical evolution, such as (Hamilton et al., 2016; Chen et al., 2022), and the aim of it is to model more precise temporal information of words. But we are also interested in your suggestion. In our future work, we will explore the method of training our model over the entire dataset.\n\nQ11: Still for section 3.1, my understanding of the motivation of Eq(2) is to model language evolution. It will be good if you can add some explanation of the motivation behind this.\n\nA11: Eq(2) denotes a learning objective to minimize the distance between two vectors by solving the matrix Q via SVD.\n\nQ12: You can try refine table 4 to save some space.\n\nA12: Thanks for your comment. We will re-forrmat the manuscript.\n\nQ13: Again, to mitigate potential ambiguities upon reading this submission, I suggest to refine the content or add specific details based on the details provided in my earlier question section, especially in cases where clarity indeed is lacking.\n\nA13: Thanks for your comment. We will follow your suggestions to revise our manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The task studied in this work is interesting and is useful for researchers seeking a deeper understanding of the history of humanity.\n\n2. It raises valuable discussion on the preservation of time-sensitive information within the framework of language modeling. \n\n3. The proposed method can address the raised issues well, and also showing a good improvement compared to existing methods.", "reasons_to_reject": "1. The study, along with the employed baseline comparisons, centers exclusively around Chinese corpus, thereby confining the applicability of the proposed method.\n\n2. The absence of a comparison with Large Language Models such as the GPT models or Chinese Language Models might be a concern.", "questions_to_authors": "I have several questions concerning both the method and experiment parts, hoping the author can clarify or make revisions in subsequent versions to enhance the clarity for future readers:\n\n1. What is Eblock in Eq(1) and how it is represented?\n\n2. In Figure (1), how is the learned h_i is used in document modeling? How are the q, k, v built? What is b_k?  By saying \"temporal features are incorporated into each layer\" in Section 3.3, do you mean h_i?\n\n3. In ablation study, by \"w/o\" temporal attention, do you mean remove L_CLS in Eq(6)? If that is the case, how do you generate the prediction y with removing the classification output? If not, can you be more specific what does this w/o mean?\n\n4. In Parameter setting, it's mentioned that training contains 30 sentences while this is not aligned with the dataset showed in Table 1. What are the training sentences here?\n\n5. How is the weights used in 3.1 initialized? \n", "paper_topic_and_main_contributions": "Automatic text dating is to predict when a document was written based on its content. This work address the existing works' limitation in capturing time-aware word and document representations with language modeling apporach. \n\nTo be specific, it propose to learn a time-aware and context-aware word representation and use it to hierarchically build document representation. The authors conduct experiments on Chinese historical corpus, showing a good improvement comparing with existing method in this task, showing the effectiveness of the proposed method.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "\nI also have several revise suggestions for the author to consider:\n\n1. I understand the limitation of this work is focusing on Chinese historical text dating. It would be beneficial to expand the conversation to encompass potential applications in other languages and provide insights based on the distinct characteristics of each language.\n\n2. Readers will be interested in the public LLM-based performance which may also be considered as baselines.\n\n3. For the representation learned in Section 3.1, by splitting the data into different collections, it's making the data more sparse.  I wonder if it is a better way to pretrain the model on the entire dataset, followed by a subsequent training within each collection.\n\n4. Still for section 3.1, my understanding of the motivation of Eq(2) is to model language evolution. It will be good if you can add some explanation of the motivation behind this.\n\n5.  You can try refine table 4 to save some space.\n\n6. Again, to mitigate potential ambiguities upon reading this submission, I suggest to refine the content or add specific details based on the details provided in my earlier question section, especially in cases where clarity indeed is lacking. "}}, {"review_id": "3u0xJ5qYD8", "reviewer": "Reviewer_8wH2", "summary": "", "questions": "Line 420-423 mentioned that \"In addition, this study does not involve external language knowledge, therefore only the structural components of pre-trained language models are used without loading pre-trained model parameters\". Do you mean that only the Transformer architecture is used but the parameters are randomly initialized? It seems from Figure 1 that you do need a pretrained model (BERT).", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Q1: The novelty of the method is low. The method for aligning word representations is directly borrowed from Hamilton et al. (2016). The hierarchical document modeling module is very similar to Grail et al. (2021).\n\nA1: There are two main contributions in this paper, as stated in the first section. One is that we propose an approach to learn temporally consistent word representations with those documents they belong to, where both temporal and contextualized information of words are captured to build distributed representations for diachronic document modeling. Here we employ a word representation alignment algorithm(Hamilton et al.) to map words of different time periods into the same semantic space, to reduce the learning cost. But the alignment itself is neither the main contribution nor the key point in this paper.\n\nThe other is that we propose a time-aware language model for historical texts, where the model can automatically find the appropriate time-specific variant of a word that is temporally consistent with the document it belongs to. Here we build a hierarchical method to model historical documents, which is also applied in many NLP tasks to model long documents. But the hierarchical method itself is neither the main contribution nor the key point in this paper.\n\nQ2: Only one dataset is used for evaluation, which does not seem sufficient to validate the effectiveness of the method.\n\nA2: Thanks for your comments. We will validate our method on more datasets and report it in the revised manuscript.\n\nQ3: (Not a major point to reject) The presentation of the paper is overall subpar. The formulas seem to be inserted as figures, and the arrangement of texts is weird here and there (e.g., the way that the numbers and names displayed in Table 2 really makes it hard to read). The authors should consider using LaTex template for formatting.\n\nA3: Thanks for your comments. We will re-format our manuscript with the latex template.\n\nQ4: Line 420-423 mentioned that \"In addition, this study does not involve external language knowledge, therefore only the structural components of pre-trained language models are used without loading pre-trained model parameters\". Do you mean that only the Transformer architecture is used but the parameters are randomly initialized? It seems from Figure 1 that you do need a pretrained model (BERT).\n\nA4: \"In addition, this study does not involve external language knowledge, therefore only the structural components of pre-trained language models are used without loading pre-trained model parameters\". It means that the pre-trained models for performance comparison, such as BERT and Hierarchical BERT, are trained only by using the dataset built in this paper, and the aim is to explore the contribution of the temporal adaptation method in a fair environment. The BERT in Figure 1 is not a comparison system. We use it to get word representations by conducting a self-supervised learning procedure.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The problem studied in the paper is interesting and not commonly seen in the NLP venues.\n* The method design is reasonable.", "reasons_to_reject": "* The novelty of the method is low. The method for aligning word representations is directly borrowed from Hamilton et al. (2016). The hierarchical document modeling module is very similar to Grail et al. (2021). \n* Only one dataset is used for evaluation, which does not seem sufficient to validate the effectiveness of the method.\n* (Not a major point to reject) The presentation of the paper is overall subpar. The formulas seem to be inserted as figures, and the arrangement of texts is weird here and there (e.g., the way that the numbers and names displayed in Table 2 really makes it hard to read). The authors should consider using LaTex template for formatting.", "questions_to_authors": "Line 420-423 mentioned that \"In addition, this study does not involve external language knowledge, therefore only the structural components of pre-trained language models are used without loading pre-trained model parameters\". Do you mean that only the Transformer architecture is used but the parameters are randomly initialized? It seems from Figure 1 that you do need a pretrained model (BERT).", "paper_topic_and_main_contributions": "This paper studies automatic text dating (ATD) by presenting a time-aware text encoder named TALM, which is aligned and adapted to represent temporal information. The paper also proposes a diachronic document modeling framework to obtain a document-level representation for long historical texts. \n\nThe authors conduct experiments on one dataset and the proposed method demonstrates improvements over selected baselines.", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Line 285: modal -> module"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Overall, the paper is acknowledged for its technical soundness, clear explanations, and insightful interpretation of results. The reviewers suggest providing more details about the dataset, exploring the use of meta-temporal features, addressing ethical and societal implications, and expanding the discussion to encompass other languages.
Decision: Accept-Findings