SlowGuess/ACL_2020_OCR huggingface dataset:
Our results show that the Wikipedia enrichment is an essential component of the protocol. For the WOTD dataset, all models exhibit a statistically significant improvement in ordering when adding CI, with the smallest improver being the LSTM classification model, with a  $+0.053$  change in  $\tau$ , and the largest improver being the BOE regression model, with a change  $+0.098$  in  $\tau$ .
For the OTD dataset, the LSTM model showed a modest but statistically significant improvement when adding CI. The BoE model presents a minor decrease in performance; however, we obtain a statistically significant improvement of  $+0.027$  in  $\tau$  by restricting the CI to only include year mentions.[11] As the OTD and the extracted CI are from different domains, the words of the contextual information most probably add too much noise for the BoE model to handle, which is why a performance improvement is observed when only including years, which are not domain-specific. This indicates that leveraging CI is important, even in this more challenging scenario, where the training data is large and the CI is from another domain, but also suggests that additional improvements, such as using domain adaptation techniques (Ziser and Reichart, 2017) for bridging the domain difference, are required to obtain better performance.
One difference between the regression and classification settings is that the latter has higher exact match metrics than the former. This reflects the nature of the two architectures: when using L1/L2 regression, the loss is proportional to the difference in the prediction, whereas in classification what matters is the probability assigned to the exact year.
On the whole, the LSTM model produces better predictions than the BOE model, according to most measures. This is perhaps unsurprising, as it is able to capture word context when analyzing the inputs, leading to more effective reasoning.
This experiment gave the following results: KT - 0.615 ± 0.002, EM - 10.8 ± 0.2, 20Y - 67.6 ± 0.3, 50Y - 84.4 ± 0.2, MAE - 36.7 ± 0.4.
7491
Dataset	CI use	Mode	Model	Accuracy	MAE
KT	EM	20Y	50Y
WOTD	No CI	Regression	BoE	0.564 ± 0.008	0.2 ± 0.1	16.8 ± 1.1	40.8 ± 1.5	83.6 ± 2.3
LSTM	0.688 ± 0.004	2.2 ± 0.6	42.7 ± 1.4	68.8 ± 0.3	51.6 ± 0.5
Classification	BoE	0.627 ± 0.007	10.9 ± 0.5	49.8 ± 0.6	66.3 ± 0.5	59.3 ± 1.0
LSTM	0.639 ± 0.021	3.5 ± 0.5	39.4 ± 1.1	61.6 ± 1.6	64.5 ± 2.3
Baseline	MEAN	0.005	0.3	9.40	28.2	127.2
CI	Regression	BoE	0.662 ± 0.008	1.9 ± 0.4	51.4 ± 1.0	67.4 ± 0.4	53.4 ± 1.0
LSTM	0.767 ± 0.008	2.1 ± 0.8	51.5 ± 4.2	77.0 ± 2.0	38.4 ± 2.0
Classification	BoE	0.705 ± 0.009	9.1 ± 0.6	61.9 ± 0.9	74.8 ± 0.5	45.8 ± 0.7
LSTM	0.692 ± 0.010	4.1 ± 0.4	44.6 ± 1.9	65.9 ± 1.6	56.9 ± 2.2
Baseline	CIYEAR	0.551	13.1	56.2	66.7	64.5
OTD	No CI	Classification	BoE	0.588 ± 0.003	11.1 ± 0.2	65.1 ± 0.2	83.1 ± 0.3	38.2 ± 0.4
Regression	LSTM	0.683 ± 0.007	3.0 ± 0.3	67.8 ± 0.8	87.3 ± 0.3	29.0 ± 0.4
Baseline	MEAN	0.006	0.5	12.9	37.4	85.7
CI	Classification	BoE	0.560 ± 0.005	10.0 ± 0.2	64.4 ± 0.3	82.3 ± 0.2	40.3 ± 0.8
Regression	LSTM	0.707 ± 0.005	3.2 ± 0.2	70.5 ± 0.6	88.9 ± 0.2	26.7 ± 0.6
Baseline	CIYEAR	0.323	6.3	41.7	62.5	60.1
Table 3: Comparison of the BoE and LSTM models under classification (Classification) and regression (Regression) settings, with and without contextual information (CI) on the WOTD dataset (top), along with results from best BoE and LSTM models on OTD dataset (bottom), with and without contextual information. We include a  $95\%$  confidence interval of the mean of each metric computed over 6 runs. Best models on the OTD dataset were picked from the WOTD development set. We also include our baseline scores (Baseline).  $\mathrm{KT} =$  Kendall's Tau, EM  $=$  Exact Match,  $20\mathrm{Y} =$  Distance under  $20\mathrm{Y}$ $50\mathrm{Y} =$  Distance under  $50\mathrm{Y}$  MAE  $=$  Mean Absolute Error.
Ablation study. Table 4 presents the results of two ablation studies on the best performing models on the WOTD development set, which are the LSTM regressor and BOE classifier. Both studies are conducted on the WOTD development set. To save space, we omit confidence intervals, but a table including those can be found in the appendix.
Study A was conducted only on datapoints from the WOTD dataset with contextual information. We observe that for both models, the impact of removing the event text and using only the extracted contextual information leads to a  $\tau$  change of  $-0.043$  for BoE and  $-0.071$  for LSTM. This shows that the heuristics we propose for extracting CI are effective at retrieving relevant information.
Study B was conducted on all datapoints from the WOTD dataset. We report the impact of removing tokens denoting years, dates and numbers from both the CI and the event text. We remove years using the method described in §4.1. We remove dates by removing any tokens within a DATE entity. We remove numbers using the like_num property of the spaCy tokenizer, which includes different forms that may be considered numerical (e.g. "1" and "one"). Clearly, the removal of dates subsumes the removal of years, and we expect the removal of numbers to remove at least part of all dates, including years, alongside other date-unrelated numbers.
The removal of these features has a very similar impact. This is particularly the case for the LSTM model, where the change in
$\tau$  was  $-0.041$ ,  $-0.042$  and  $-0.051$  when removing years, dates and numbers, respectively. BoE presents similar differences in performance when removing those features, with a change in  $\tau$  of  $-0.045 / - 0.031 / - 0.054$  when removing years/dates/numbers. These results support our hypothesis that substantial information about the time of an event is encoded in the vocabulary used, and not only in the time expressions.
Human Performance We compare our results to human performance on this task. Three participants were given 100 randomly selected events from the WOTD dataset and were asked to predict years of occurrences, without using any contextual information. All participants consider themselves as having good knowledge of history, but are not history experts. On average, their error was 52.3 years. The participant who had the best results had a mean error of 34.6 years, which is only 3.8 years less than our best result on the WOTD dataset.
In order to demonstrate the challenges put forth by the addressed task, we examine some events from the OTD development dataset on which our best performing models, LSTM regressor and BoE classifier, got significant prediction errors.
We observe that some events contain words that are usually associated with a different period in time than the year the event occurred in. For exam
Model	Accuracy
KT	EM	20Y	50Y	MAE
A	BoE	0.674	7.9	49.6	63.1	48.2
– event text	0.631	7.0	48.2	60.6	54.6
LSTM	0.765	1.8	50.2	78.8	39.0
– event text	0.694	1.3	39.9	69.7	50.6
B	BoE	0.668	9.1	55.3	71.0	50.7
– years	0.623	7.4	46.1	65.0	61.1
– dates	0.637	7.4	46.7	64.2	60.5
– numbers	0.614	8.0	46.3	64.2	63.1
LSTM	0.774	1.8	50.4	77.3	39.9
– years	0.733	1.3	41.0	68.5	48.3
– dates	0.732	1.4	42.5	70.1	47.7
– numbers	0.723	1.3	40.1	68.8	49.1
Table 4: Ablation study for BOE and LSTM models. Study A was conducted only on datapoints from the WOTD dataset with contextual information, and we report the impact of removing the event text from both models. Study B was conducted on all datapoints from the WOTD dataset, and we report the impact of removing tokens denoting years, dates and numbers. Tokens of these types were removed from the event text and from the contextual information.  $\mathrm{KT} =$  Kendall's Tau,  $\mathrm{EM} =$  Exact Match,  $20\mathrm{Y} =$  Distance under  $20\mathrm{Y}$ ,  $50\mathrm{Y} =$  Distance under  $50\mathrm{Y}$ , MAE  $=$  Mean Absolute Error.
ple, “Portuguese expel Jesuits” occurred in 1911, but most Jesuits-related events in our training data occurred in the 16th century. One of these events which is particularly similar to the above is “English parliament expels Jesuits”, which is dated to 1584. Probably for these reasons the LSTM and BoE had similar outputs for this event – 1559 and 1581, respectively. Another example for such an event is “Order of Merit instituted by King Edward VII”, which occurred in 1902, but the word “King” normally appears in events dated to earlier centuries. The LSTM model output for this event is 1527, and BoE model output is 1639. Both events had no CI extracted for them, therefore the models had to rely on words in the event description only.
An example for which relevant CI was extracted but the models still erred substantially is the event "All female jury hears case of Judith Catchpole accused of killing her child (acquit her) in Patuxent County, Maryland". This event is dated to 1656, but the BOE model prediction for the event is 1957, and the LSTM model prediction, 1873, is only slightly better. The contextual information extracted for this event was "Upon her arrival she was accused of several crimes, resulting in a trial on September 22, 1656 in the General Provincial Court in Patuxent County, Maryland". The exact date of occurrence does appear in the extracted data, and still both
models have a substantial prediction error. This is probably due to the fact that our training data contains many "court" and "jury" related events, where most events containing "court" are relatively recent (19th century and later), and almost all "jury" related events are dated after 1900.
In some cases, the extracted CI can mislead our models. For the event "Scotland and France form an alliance, the beginnings of the Auld Alliance, against England" that occurred in 1295, LSTM predicted the year 1659. Five sentences were extracted for this event, which contained the years 1603 and 1707. Another example is "Over 250 years after their deaths, William Penn and his wife Hannah Callowhill Penn are made Honorary Citizens of the United States" occurred in 1984. The CI extracted includes the exact true date of the event, but also includes information regarding the Penns' lives, and contains years ranging between 1680 to 1726. This is probably the cause of error for the BOE model, which predicts the year 1721, whereas the LSTM model may have been able to better filter the correct CI, and predict the year 1921.
Errors can also arise from terms that are ambiguous between time periods. "Queen Elizabeth" is such a term: it can indicate an event from the 16th century, but also an event from the 20th/21st centuries. Indeed, we notice confusion of the BOE model on events related to Queen Elizabeth. For example, "Francis Drake knighted by Queen Elizabeth I aboard Golden Hind at Deptford" occurred in 1581, but the BOE model predicts the year 2013 - even though the true target year appears in the extracted CI for the event: "I visited the royal dockyard on 4 April 1581 to knight the adventurer Francis Drake." Similarly, the event "Ted Hughes is appointed British Poet Laureate by Queen Elizabeth II" occurred in 1984, but the BOE model predicts the year 1579, which corresponds to Queen Elizabeth I. We note that for those two events the LSTM model gave better predictions (1566 for the first event and 1981 for the second), which may be related to the inherent difficulty of BOE to address multi-word expressions like "Queen Elizabeth I".
Work on event ordering can largely be categorized into event ordering in context, which aims to order event instances within a given text or discourse and is tackled as part of the TempEval shared tasks (UzZaman et al., 2013), and lexical event order-
7493
ing (Abend et al., 2015), which attempts to order event types by their prototypical temporal order. Somewhat in between these lines of work is cross-document event ordering (Minard et al., 2015), which orders events that are mentioned across different documents. However, this task does not rely on machine reading external textual resources as we do here, and does not focus on historical events that by their nature are described in a variety of (often incompatible) ways.
A related line of work to ours was pursued in the context of information retrieval (IR). Jatowt et al. (2013) tackled the task of estimating what the "focus time" of a given document is. Focus time is defined as the time to which the main event addressed by the document refers to. They do so by computing the association of words and time expressions, based on their co-occurrence, using a bag-of-words method.
Das et al. (2017) address the task of focus time prediction for short event descriptions, which resembles the task at hand. They do so by using cosine similarity to rank a set of candidate years for each event, all of which are computed using word embeddings. In a similar vein, Morbidoni et al. (2018) find the focus time of short event descriptions by relying on year mention statistics in related Wikipedia articles and DBpedia entries.
While these two works are related to our task in spirit, our work is not an instance of the event focus time (EFT) task. In fact, we believe the EFT task can be seen as a special case of the HEO task. This is evidenced by the approach of EFT systems, which exhibits traditional IR design and techniques, such as producing a ranking of candidate predictions for each document, and is evaluated using ranking-specific metrics that forbid system designs such as predicting years using regression. As HEO subsumes EFT, we attempted to evaluate the performance of EFT systems in the HEO task, but have been unable to obtain code for either of the systems. We have also been unable to reimplement the systems: (Das et al., 2017) leaves implementation details unspecified, and (Morbidoni et al., 2018) utilizes a proprietary system.
Another related line of work seeks to create timelines of temporal events by predicting their starting and ending points. McClosky and Manning (2012) address the problem of ensuring semantically consistent timelines by finding patterns in the ordering of endpoints of different event types, which adds a
common sense reasoning component to the system. Leeuwenberg and Moens (2018) construct a relative timeline of events directly, which allows them to circumvent typical pitfalls of pair-wise classifiers, such as computationally intractable inference and constructing globally inconsistent orderings (with cycles). Our work takes a similar approach but instead is able to construct an absolute timeline for the restricted domain of historical events.
Within the domain of temporal text understanding, the extraction and normalization of temporal expression may inform the task at hand. For example, Kuzey et al. (2016) defined the task of tagging temporal expressions, which are named events or facts with temporal scope, such as "second term of Angela Merkel". They used a rule-based system to detect such expressions in free-text and map these expressions to a knowledge base (KB) containing time scopes of temporal events and facts. This approach requires the existence of KB records containing time scopes for the events.
In this paper we argued that the task of predicting the time of historical events strikes a balance between being a focused task, with transparent evaluation and interpretable results, and presenting challenges that are not simple to overcome using standard NLP models. We outlined a procedure to extract the CI related to an event and compared two approaches for the task, using bag of embeddings and an LSTM, showing that the latter achieves the best performance. Future work will explore the use of domain adaptation techniques to enhance performance where the domains of the CI and event text differ substantially.
We thank the anonymous reviewers for helpful feedback. We would also like to thank Maximin Coavoux, Simone Teufel, and Ryan Cotterell for their help and comments. We gratefully acknowledge the support of Bloomberg (Cohen). This work was partially supported by the Israel Science Foundation (grant No. 929/17)
Omri Abend, Shay Cohen, and Mark Steedman. 2015. Lexical event ordering with an edge-factored model. In Proc. of NAACL, page 1161-1171.
Dzmitry Bahdanau, KyungHyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proc. of ICLR.
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2016. Enriching word vectors with subword information. CoRR, abs/1607.04606.
Razvan Bunescu and Marius Paşa. 2006. Using encyclopedic knowledge for named entity disambiguation. In Proc. of EACL, pages 9-16.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457.
Henry E. Daniels. 1944. The relation between measures of correlation in the universe of sample permutations. Biometrika, 33(2):129-135.
Supratim Das, Arunav Mishra, Katsumi Tanaka, and Vinay Setty. 2017. Estimating event focus time using neural word embeddings. In Proceedings of the 26th ACM International Conference on Information and Knowledge Management, CIKM, pages 2039-2042.
Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson F. Liu, Matthew Peters, Michael Schmitz, and Luke Zettlemoyer. 2018. AllenNLP: A deep semantic natural language processing platform. In Proceedings of Workshop for NLP Open Source Software (NLP-OSS), pages 1-6, Melbourne, Australia. Association for Computational Linguistics.
Alex Graves, Santiago Fernández, and Jürgen Schmidhuber. 2005. Bidirectional LSTM networks for improved phoneme classification and recognition. In Proceedings of the 15th International Conference on Artificial Neural Networks: Formal Models and Their Applications - Volume Part II, ICANN'05, pages 799-804, Berlin, Heidelberg. Springer-Verlag.
Dhruv Gupta and Klaus Berberich. 2016. Diversifying search results using time. In Advances in Information Retrieval, pages 789-795, Cham. Springer International Publishing.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2015. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. CoRR, abs/1502.01852.
Karl Moritz Hermann, Tomáš Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. arXiv:1506.03340 [cs]. ArXiv: 1506.03340.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Comput., 9(8):1735-1780.
Adam Jatowt, Ching-Man Au Yeung, and Katsumi Tanaka. 2013. Estimating document focus time. In Proceedings of the 22Nd ACM International Conference on Conference on Information and Knowledge Management, pages 2273-2278.
Maurice G. Kendall. 1938. A new measure of rank correlation. Biometrika, 30(1/2):81-93.
Maurice G. Kendall. 1945. The treatment of ties in ranking problems. Biometrika, 33.
Diederik P. Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. CoRR, abs/1412.6980.
Erdal Kuzey, Vinay Setty, Jannik Strötgen, and Gerhard Weikum. 2016. As time goes by: Comprehensive tagging of textual phrases with temporal scopes. In WWW '16 Proceedings of the 25th International Conference on World Wide Web, pages 915-925.
Artuur Leeuwenberg and Marie-Francine Moens. 2018. Temporal information extraction by predicting relative time-lines. CoRR, abs/1808.09401.
Xue Li. 2018. Temporal ordering of historical events. Master's thesis, University of Edinburgh.
David McClosky and Christopher D. Manning. 2012. Learning constraints for consistent timeline extraction. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 873-882, Jeju Island, Korea. Association for Computational Linguistics.
Anne-Lyse Minard, Manuela Speranza, Eneko Agirre, Itziar Aldabe, Marieke van Erp, Bernardo Magnini, German Rigau, and Ruben Urizar. 2015. Semeval-2015 task 4: Timeline: Cross-document event ordering. In 9th International Workshop on Semantic Evaluation (SemEval 2015), pages 778-786.
Mike Mintz, Steven Bills, Rion Snow, and Dan Jurafsky. 2009. Distant supervision for relation extraction without labeled data. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP: Volume 2-Volume 2, pages 1003-1011. Association for Computational Linguistics.
Christian Morbidoni, Alessandro Cucchiarelli, and Domenico Ursino. 2018. Leveraging linked entities to estimate focus time of short texts. In Proceedings of the 22Nd International Database Engineering & Applications Symposium, IDEAS 2018, pages 282-286, New York, NY, USA. ACM.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532-1543. Association for Computational Linguistics.
7495
Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proc. of NAACL.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad:  $100,000+$  questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383-2392.
Gabriel Stanovsky and Ido Dagan. 2016. Creating a large benchmark for open information extraction. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2300-2305.
Naushad UzZaman, Hector Llorens, Leon Derczynski, James Allen, Marc Verhagen, and James Pustejovsky. 2013. Semeval-2013 task 1: Tempeval-3: Evaluating time expressions, events, and temporal relations. In *SEM-SemEval'13, pages 1-9.
Yftah Ziser and Roi Reichart. 2017. Neural structural correspondence learning for domain adaptation. In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pages 400-410.
A Key Entities Extraction
We introduce the full set of rules for key entity extraction. The following are considered key entities:
1. Named entities, excluding the following labels: MONEY, TIME, PERCENT, DATA, ORDINAL, QUANTITY, CARDINAL.
2. A compound whose head, or the head of its head, is the root of the sentence. For example, in the event: "Apollo program: Apollo 14 returns to Earth after the third manned Moon landing.", this rule will extract "Apollo program".
3. An adjectival modifier or nominal modifier whose head is the root, where the root is not a verb. For example, in the event "Mexican-American War: The first large-scale amphibious assault in U.S. history is launched in the Siege of Veracruz.", this rule will extract "Mexican-American War".
4. All nominal subjects, except pronouns and nominalized adjectives. An example for this rule can be found in the paper itself.
5. All passive nominal subjects that are proper nouns. For example, in the event "The USS George Washington is launched. It is the first
nuclear-powered ballistic missile submarine." this rule extracts "USS George Washington".
6. All direct objects that are proper nouns.
We note that for all rules except the first we take all the sub-tree to which the word we found belongs. We remove from the sub-tree determiners, punctuation and adverbs. In the example given above – “The USS George Washington is launched. It is the first nuclear-powered ballistic missile submarine.” – the word that complies to the fifth rule above is “Washington”, but we extract “USS George Washington”. In addition, we remove relative clauses of the phrase. For example, in the event “The British Parliament passes the Stamp Act that introduces a tax to be levied directly on its American colonies,” we are interested in extracting “Stamp Act”, but we leave out the part “that introduces a tax...”. Similarly, from the event “The debut exhibition of the Belitung shipwreck, containing the largest collection of Tang dynasty artifacts found in one location, begins in Singapore,” we leave out the part “containing the largest...” when extracting “debut exhibition of Belitung shipwreck”.
B Extended Ablation Results
Refer to Table 5 for an expanded table of the ablation experiments that includes metric error.
7496