SlowGuess/ACL_2020_OCR huggingface dataset:
Or Honovich $^{1,*}$  Lucas Torroba Hennigen $^{2,*}$  Omri Abend $^{1}$  Shay B. Cohen $^{3}$
<sup>1</sup>The Hebrew University of Jerusalem <sup>2</sup>University of Cambridge <sup>3</sup>University of Edinburgh
or.honovich@gmail.com lft26@cam.ac.uk
oabend@cs.huji.ac.il scohen@inf.ed.ac.uk
Machine reading is an ambitious goal in NLP that subsumes a wide range of text understanding capabilities. Within this broad framework, we address the task of machine reading the time of historical events, compile datasets for the task, and develop a model for tackling it. Given a brief textual description of an event, we show that good performance can be achieved by extracting relevant sentences from Wikipedia, and applying a combination of task-specific and general-purpose feature embeddings for the classification. Furthermore, we establish a link between the historical event ordering task and the event focus time task from the information retrieval literature, showing they also provide a challenging test case for machine reading algorithms.
Machine reading concerns the extraction of entities and relations from text and the ability to use them meaningfully, for instance by answering questions based on them, inferring other relations from them, or using them to compile knowledge bases. Such an inclusive task definition necessarily builds on a wide range of NLP capabilities, from syntactic and semantic analysis, to the use of world knowledge and common sense. The inclusive nature of the task supports the development of general-purpose methods, but also results in low performance in absolute terms, difficulty in defining widely agreed-upon evaluation protocols, and difficulties identifying the sources of prediction errors (Stanovsky and Dagan, 2016; Rajpurkar et al., 2016; Clark et al., 2018).
This paper addresses a sub-task of machine reading, namely the task of estimating when a historical
Year	Event text
OTD	2005	107 die in Amagasaki rail crash in Japan.
1939	BMI (Broadcast Music Incorporated) formed.
1864	General Sherman's armies reach Savannah & 12 day siege begins.
WOTD	1887	Buffalo Bill Cody's Wild West Show opens in London.
1399	Henry IV is proclaimed King of England.
1943	First Flight of the Gloster Meteor, Britain's first combat jet aircraft.
Table 1: Entries from the OTD and WOTD datasets.
event took place. This distinguishes it from traditional question answering (Rajpurkar et al., 2016) as the answer may not be given in the text but the models should still be able to place events in the correct period of time. In turn, this means that models trained for historical event ordering may have real-world applications such as to serve as a fallback for temporal question answering when the answers are not present in the text and to improve search engines that leverage the implicit time of queries (Gupta and Berberich, 2016).
Concretely, given a short text description of a historical event, and an external data source (henceforth contextual information or CI), the task is to predict the year in which the event happened. The external source in our case is Wikipedia. For example, given the event description "The Government of Turkey expels Patriarch Constantine VI from Istanbul," the task is to infer the year it took place (i.e., 1925). We select Wikipedia as a source for contextual information, due to its broad coverage, and the wide interest it receives in the NLP community. Indeed, Wikipedia has often featured as a semi-structured knowledge base, e.g., as a source of concept grounding (Bunescu and Paşa, 2006) and indirect supervision (Mintz et al., 2009).
We hypothesize that aside from time expressions, the CI words themselves give an approximate time in which an event happened. For example, the pres
$^{1}$ Code and data are available at https://github.com/ltorroba/machine-reading-historical-events. *Equal contribution.
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7486-7497 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
ence of the word "spacecraft" in the CI probably indicates an event that occurred after 1900, while the presence of the word "sword" most likely indicates an event that occurred before 1900. The task is therefore different from tasks addressing the extraction and normalization of time expressions, or from related tasks pursued in the context of information retrieval (see §8). Our results support this hypothesis, and demonstrate that even when time expressions are not present in the data, it is still possible to predict the approximate year in which an event happened.
We compile two datasets for the task, based on the websites "Wikipedia On This Day" webpages (WOTD) and "On This Day" (OTD). We consider WOTD as an in-domain setting, given that it is taken from Wikipedia as well (albeit from an entirely disjoint part of Wikipedia). The OTD setting was selected to be maximally challenging for leveraging external data sources, since (1) event descriptions are taken from a different website, and may be formulated very differently from Wikipedia; (2) it is an order of magnitude larger, and so the classifier has plenty of data to train on, even without relying on external data sources.
Our results show that on WOTD, good performance can be obtained by detecting relevant sentences from Wikipedia and extracting year mentions in them, but that substantially better performance can be reached when additionally encoding the entire sentences, using neural machinery. In OTD, CI yields more modest improvements. Results in absolute terms are high: the best models obtain a mean Kendall's  $\tau$  correlation with the correct event ordering of 0.77 (WOTD) and 0.71 (OTD).
2 Task Definition and Motivation
The historical event ordering (HEO) task is defined as follows. Given a set of brief event descriptions and some textual resource, the task is either to predict the year in which each event occurred, or to find a ranking of events such that they are ordered by date of occurrence. The first variant is stronger than the second, as it implies a ranking. Our evaluation uses both rank correlation (Kendall's  $\tau$ ) and measures of the distance between the year the event took place and the predicted year. See Section 5.2 for details.
Differences from Question Answering. While traditional question answering tasks require the answer to be in the text (e.g., Hermann et al., 2015; Rajpurkar et al., 2016), the HEO task is based on
estimating the time of occurrence of an event. This estimation is based solely on lexical cues, and does not require an explicit answer in any text. This is a major advantage of HEO models, as explicit answers are not always present in the text for two reasons: (i) we would need a massive amount of text for good coverage of historical events, which may be unfeasible to use in real-world applications; and (ii) new events are constantly occurring, and existing machine reading comprehension models will invariably fail on those (e.g., "When was Donald Trump elected president?" will not be covered in old data, but could be inferred to have happened recently based on recognizing the named entity "Donald Trump"). As answers are not guaranteed to be in the text, the HEO task is somewhat more challenging than traditional question answering tasks. The task's challenge is also evidenced in that it requires temporal commonsense reasoning and in being challenging for humans (see §6).
Real-World Applications. As previously mentioned, HEO models do not assume the presence of the answer in the source text, and can thus be used for temporal question answering when the answers are not present in it. By leveraging the lexical information that exists only in the question itself, these models can serve as a fallback for such cases. Other possible applications are dating of historical documents based solely on the documents' text, improving search engines that leverage the time of queries (Gupta and Berberich, 2016), as well as making inferences that involve rough temporal placement of the statement (e.g., inferences involving refrigerators are unlikely to be relevant before the 20th century).
This work introduces two datasets: WOTD and OTD. Despite the similarity in their names, we are not aware of any influence or other relation between them. Using both datasets thus makes our experimental analysis less prone to be biased by dataset-specific artifacts.
Wikipedia On This Day (WOTD) was scraped from Wikipedia's On this day webpages. The dataset contains 6,809 entries. Some example entries are presented in Table 1. Events in Wikipedia's On This Day pages are crowdsourced,
$^{2}$ E.g., https://en.wikipedia.org/wiki/Wikipedia:On.this_day/Today, accessed 03/2018.
Figure 1: Distribution of event years in the OTD and WOTD, binned into bins of 100 years. The y-axis corresponds to the proportion of events falling into the bin.
but must adhere to specific guidelines<sup>3</sup> which include the validity and overall relevance of the historical event. The earliest label in this dataset is 1302, and the latest is 2018. The median year is 1855.0 whereas the mean is 1818.7. The standard deviation is 156.5 years.
On This Day (OTD) is a scrape of the On This Day – Today in History, Film, Music and Sport (Li, 2018).<sup>4</sup> On This Day has a dedicated team that adds, verifies content, and responds to corrections from the public.<sup>5</sup> The dataset contains 75,135 entries consisting of a sentence describing the event and the event's date. We removed 96 events from the original dataset, which happened BCE (Before Common Era), and also removed events that had not happened yet. The earliest event in the dataset occurred on year 1 CE (Common Era), and the latest occurred in 2018 CE. The median label is 1960.0, while the mean label is 1913.8, so the distribution of labels is not uniform: there are more events occurring in recent times. The standard deviation for the labels is 172.3 years.
Examples of entries from OTD are presented in Table 1. We note that the overwhelming majority of events in the datasets are real historical events, and though we did not conduct an exhaustive analysis, the only two we identified as fictional were removed by our filters. There are 8 events that are dated in the future, and all but one of those
("Earth's 1st contact with the extra-terrestrial Vulcan species in the Star Trek universe", on 2063 CE) correspond to either calendar occurrences (e.g., "Beginning of 2nd Julian Period (1/1 OS)", on 3268 CE) or astronomical events (e.g., "Comet Swift-Tuttle approaches close to Earth", on 2126 CE). Our pruning strategy (discard events before 1 CE) was deliberately aggressive, removing 88 events including widely accepted ones (e.g., "Battle of Actium", on 31 BCE); however, it is also effective in removing potentially fictitious events (e.g., "Creation of the world begins according to the calculations of Archbishop James Ussher", on 4004 BCE) or whose exact date may not be known (e.g., "Battle of Megiddo" dated to 1457 BCE, but subject to debate).
Figure 1 shows the distribution of event years in OTD and WOTD. Both datasets have significantly more recent events from the last few centuries. We use a random  $80/10/10$  split of each dataset to form the training, validation and test sets.
4 Algorithmic Approaches
We propose two models: a bag of embeddings model (BoE) and a recurrent neural network model (LSTM). Both take a training example and output a timestamp, in our case the year of the event. We explore two supervised settings: a classification setting, where each possible year corresponds to a different class, and a regression setting, where the labels are the numerical value of the timestamp.
As baselines, we define two models: one predicts the mean year of the training set (MEAN), and one predicts the median year present in the extracted CI, falling back to the other baseline if no years are found (CIYEAR).
4.1 Retrieving Relevant Wikipedia Sentences
Key Entities And Actions. We first identify the key entities and actions in each event description. Concretely, for a given event description  $e$ , we define its key entities to be phrases from  $e$  that are likely to be the topic of a Wikipedia article that contains information relevant to  $e$ . We define key actions to be a tuple of all verbs in  $e$ , excluding some aspectual (e.g., "begin") and auxiliary verbs. We lemmatize all key actions.
For example, given the event description "The Sixth Coalition attacks Napoleon Bonaparte in the Battle of Leipzig", we mark ("Sixth Coalition",
$^{3}$ https://en.wikipedia.org/wiki/Wikipedia:Selected_anniversaries, accessed 04/2020.  
 $^{4}$ https://www.onthisday.com, accessed 01/2019.  
 $^{5}$ https://www.onthisday.com/about.php, accessed 04/2020.
"Napoleon Bonaparte", "Battle of Leipzig") as the key entities, and "attack" as the key action.
Entities and actions are extracted using a set of pre-defined rules, based on linguistic features such as part-of-speech (POS) tag, syntactic dependency labels, and entity type, for words recognized as named entities. Linguistic features, including named entities, are extracted using spaCy. Some example rules for detecting key entities are:
1. Take all named entities, excluding some entity types such as MONEY, PERCENT and ORDINAL.
2. Take all nominal subjects, except pronouns and nominalized adjectives. For example, for "The Sixth Coalition attacks Napoleon Bonaparte in the Battle of Leipzig", "Sixth Coalition" is marked as a key entity.
The complete set of rules can be found in the supplementary material. The majority of key entities are named entities and are therefore identified by the first rule above.
Article Retrieval. We use the extracted key entities to retrieve relevant Wikipedia articles. For each key entity, we retrieve the first search result returned for the entity name, as proposed by the Wikipedia API. We use the Python Wikipedia library for performing the queries.
Sentence Filtering. Filtering seeks to identify sentences related to the historical event in question. For example, for the event "The Skye Bridge is opened", the sentence "Construction began in 1992 and the bridge was opened by Secretary of State for Scotland Michael Forsyth on 16 October 1995" from the article "Skye Bridge" is relevant.
We denote by  $\{t_1,\dots ,t_k\}$  the key entities for each event, where  $k$  varies from one event to another, and test several filtering methods:
1. Sentences from an article with title  $t_i$  that contain one or more  $t_j$  for  $j \neq i$ , and a key action.
2. Sentences from an article with title  $t_i$  that contain one or more  $t_j$  for  $j \neq i$ .
3. Sentences from an article with title  $t_i$  that contain all  $t_j$  for  $j \neq i$ .
4. Sentences that contain a date.
5. Sentences from an article with title  $t_i$  contain one or more  $t_j$  for  $j \neq i$ , and a date.
Article sections with headers such as "See also" and "Bibliography" are removed.
Following a manual inspection of the extracted sentences with each of the methods, we find the following method works best: (1) find all sentences according to the first filter; (2) if no relevant sentences are found, apply the second filter instead. In addition, we add the original textual description of the event (taken from OTD/WOTD) to the list of relevant sentences.
Extracting Year Mentions. Given the relevant sentences for each event, we extract from them all year mentions. Years are extracted using the following method: first, we use named entity recognition to extract all dates. Second, of the words recognized as dates, we keep only those whose POS tag is NUMBER.<sup>9</sup> We then parse the dates and extract years, using a simple rule-based parser.<sup>10</sup> We present here some statistics regarding years extracted for the WOTD validation set. For  $1.8\%$  of the events, the real year appeared in the event title itself. For  $59.5\%$  of the events, at least one year appeared in the contextual information extracted from Wikipedia. Out of the events for which at least one year was extracted,  $59.5\%$  had the correct year in the extracted information. In total, for  $35.4\%$  of the events, the correct year appeared in the contextual information extracted.
To obtain an estimate of the difficulty of the task, we design two baseline models. The MEAN model predicts the mean year seen in the training set, adding Gaussian noise  $\epsilon \sim \mathcal{N}(0,1yr)$  to break ties and induce an ordering. The CIYEAR model extracts year mentions, as detailed above, and predicts the median of all extracted years entities. If no years are found, the model defaults to MEAN.
4.3 Bag of Embeddings Model
We use two types of features: (1) the average of the word embeddings for all lemmatized words in the extracted sentences, excluding stop words and punctuation (as defined by SpaCy); (2) the median value of all year mentions. To represent the median year, we use one-hot encoding for the tens,
<sup>6</sup>In some cases, overlapping entities are extracted. During the next step of extracting Wikipedia's articles, we remove duplicate articles.
7www.spacy.io. We used spaCy's v2 "en_core_web-lg" model.
$^{8}$ www.pypi.org/project/wikipedia
<sup>9</sup>Again, all linguistic features are extracted using spaCy.
10www.pypi.org/project/python-dateutil
7489
hundreds and thousands of the median year and concatenate this encoding to the average embedding. We experimented with encoding the least significant digit as well, but find this lowers results. We explore two variants of the model:
Classification. In the classification setting, the final module consists of a multilayer perceptron (MLP), where class labels are the target years. We note that in the classification settings, the predicted years can only be those that appear in the training set. Since most of our evaluation metrics do not require an exact prediction, but rather an approximate prediction, the classification still yields good results. The final layer is a softmax layer, and the loss function used is log-loss.
Regression. In the regression setting, the network architecture is an MLP with a single output. The regression target is the year of occurrence. The loss function used is L1 loss. We experimented with mean squared error loss (L2) as well, but this gave lower performance.
4.4 Long Short Term Memory Model
The LSTM model takes as input the tokens for the event text and the extracted sentences. A bidirectional LSTM (Hochreiter and Schmidhuber, 1997; Graves et al., 2005) is used to compute an encoding of the event sentence (e) and each CI sentence  $(\mathbf{c}_1,\dots ,\mathbf{c}_n)$ . We then use an attention mechanism (Bahdanau et al., 2015) to compute a similarity score between the event sentence and each CI sentence, and compute an attention-weighted average of the CI encodings,  $\mathbf{c}^{\prime}$ . When training models with CI, we concatenate both e and  $\mathbf{c}^{\prime}$  and use that as input to an MLP that performs the final year prediction. When not using CI, the only input to the MLP is e. The structure of the MLP depends on whether the model is operating on a classification or a regression setting. The two variants we explore are:
Classification. In the classification setting, the final module is composed of an MLP that computes the logits of the event happening in a specific year. All years between the minimum and maximum year present in the training set are valid targets. We minimize the cross-entropy loss of the predicted year.
Regression. In the regression setting, the final module consists of an MLP with a single output.
Setting	LSTM	MLP
CI Use	Mode	L	D	L	D
WOTD	No CI	Regression	3	200	3	50
Classification	3	200	3	300
CI	Regression	2	200	3	200
Classification	2	100	1	50
OTD	No CI	Regression	3	300	2	100
CI	2	200	2	100
Table 2: Setting-specific hyperparameter values for the LSTM model.  $\mathrm{L} =$  Layers,  $\mathrm{D} =$  Layer dimensionality.
The regression target is the normalized year of the event. We normalize by subtracting the mean year of the training set and dividing the result by the standard deviation. We experimented with regression to unnormalized targets, but found this degraded performance. We minimize the L2 loss of the predicted year.
In this section we describe our experimental setup and the evaluation metrics we use.
5.1 Hyperparameters and training
For the BoE model, in the classification setting we set it to have two hidden layers, each with 1000 neurons. We ran experiments with Glove (Pennington et al., 2014) and FastText (Bojanowski et al., 2016) word embeddings and found that Glove vectors with dimension 300, pretrained on Wikipedia 2014, performed best. The initial learning rate of the MLP is set to 0.001. We use L2 regularization with  $\alpha = 10^{-4}$ . In the regression setting the model has one hidden layer with 32 units. We use Glove with dimension 300. The initial learning rate is set to 0.01. In both settings, we use ReLU as an activation function and Adam for an optimizer (Kingma and Ba, 2014). We experimented with L1 and L2 regularization but found that this doesn't improve performance.
We found the LSTM model to be sensitive to hyperparameter values, and therefore tuned it individually for each setting. The final hyperparameters are shown in Table 2. We use the Adam optimizer (Kingma and Ba, 2014) with  $\eta = 0.001$ ,  $\beta_{1} = 0.9$  and  $\beta_{2} = 0.999$ , and use PReLU activations (He et al., 2015) in the MLP. We train for a maximum of 100 epochs, doing early stopping if the validation loss has not improved in 25 epochs. Furthermore, we decay the learning rate by a factor of 0.1 if there is no reduction in validation
set loss for 10 epochs. Preliminary experiments with Glove (Pennington et al., 2014), ELMo (Peters et al., 2018) and FastText (Bojanowski et al., 2016) word embeddings showed that concatenating 200-dimensional Glove and 300-dimensional FastText embeddings performed best. We experimented with L2 regularization and dropout on both the MLP and LSTM but found that the performance improvement was negligible, and so we did not use them for our final experiments. Our LSTM implementation was done using AllenNLP (Gardner et al., 2018). All hyperparameter tuning was done against the development data.
Kendall's Tau  $(\tau)$ , formally Kendall's rank correlation coefficient (Kendall, 1938, 1945), is a standard metric used to measure two different rankings of the same set. Formally, for two rankings  $X$  and  $Y$ , the form of a general correlation coefficient (Daniels, 1944) is
$$
\tau = \frac {\sum_ {i , j = 1} ^ {n} a _ {i j} b _ {i j}}{\left(\sum_ {i , j = 1} ^ {n} a _ {i j} ^ {2}\right) \left(\sum_ {i , j = 1} ^ {n} b _ {i j} ^ {2}\right)}, \tag {1}
$$
where  $a_{ij}$  is the score given to a pair  $(X_i, X_j)$  and  $b_{ij}$  to the pair  $(Y_i, Y_j)$ . For Kendall's  $\tau$ ,  $a_{ij} = 1$  if  $X_i < X_j$  and  $a_{ij} = -1$  if  $X_i > X_j$ , and similarly for  $b_{ij}$  and  $Y$ . In plain words,  $\tau$  is the number of pairs which  $X$  and  $Y$  order in the same way minus the number of pairs that are not ordered in the same way, divided by the total number of pairs. For the case where there are no ties, Kendall's  $\tau$  is a shifted and scaled version of pairwise accuracy, where  $\tau = -1.0$  corresponds to zero accuracy and 1.0 to perfect accuracy. To accommodate for ties, we set  $a_{ij} = 0$  when  $X_i = X_j$ , and  $b_{ij} = 0$  when  $Y_i = Y_j$ , as described by Kendall (1945). This has the same effect as replacing tied members in each set with all permutations of a contiguous set of integer ranks and averaging by the total number of permutations.
Exact Match. Percentage of events in which the predicted year exactly matches the gold-standard.
Mean Absolute Error. This is the absolute mean error for the predictions, in years.
Distance under 20Y and 50Y. The percentage of events whose prediction error was under 20/50 years.
Table 3 presents the results of our experiments. We report the average of each statistic over 6 runs, alongside the standard error of the mean at  $95\%$  confidence. We include a detailed comparison of the different architectures on the WOTD dataset. We additionally select the best performing BOE and LSTM models on the WOTD development set and train them on the OTD dataset.