SlowGuess/NAACL_2021_OCR huggingface dataset:
Aman Madaan, Yiming Yang
amadaan@cs.cmu.edu
This paper presents the first study on using large-scale pre-trained language models for automated generation of an event-level temporal graph for a document. Despite the huge success of neural pre-training methods in NLP tasks, its potential for temporal reasoning over event graphs has not been sufficiently explored. Part of the reason is the difficulty in obtaining large training corpora with human-annotated events and temporal links. We address this challenge by using existing IE/NLP tools to automatically generate a large quantity (89,000) of system-produced document-graph pairs, and propose a novel formulation of the contextualized graph generation problem as a sequence-to-sequence mapping task. These strategies enable us to leverage and fine-tune pre-trained language models on the system-induced training data for the graph generation task. Our experiments show that our approach is highly effective in generating structurally and semantically valid graphs. Further, evaluation on a challenging hand-labeled, out-of-domain corpus shows that our method outperforms the closest existing method by a large margin on several metrics. We also show a downstream application of our approach by adapting it to answer open-ended temporal questions in a reading comprehension setting.
Temporal reasoning is crucial for analyzing the interactions among complex events and producing coherent interpretations of text data (Duran et al., 2007). There is a rich body of research on the use of temporal information in a variety of important application domains, including topic detection and tracking (Makkonen et al., 2003), information extraction (Ling and Weld, 2010), parsing of clinical records (Lin et al., 2016), discourse analy
sis (Evers-Vermeul et al., 2017), and question answering (Ning et al., 2020).
Graphs are a natural choice for representing the temporal ordering among events, where the nodes are the individual events, and the edges capture temporal relationships such as "before", "after" or "simultaneous". Representative work on automated extraction of such graphs from textual documents includes the early work by Chambers and Jurafsky (2009), where the focus is on the construction of event chains from a collection of documents, and the more recent CAEVO (Chambers et al., 2014) and Cogcomptime (Ning et al., 2018), which extract a graph for each input document instead. These methods focus on rule-based and statistical submodules to extract verb-centered events and the temporal relations among them. As an emerging area of NLP, large scale pre-trained language models have made strides in addressing challenging tasks like commonsense knowledge graph completion (Bosselut et al., 2019) and task-oriented dialog generation (Budzianowski and Vulić, 2019). These systems typically fine-tune large language models on a corpus of a task-specific dataset. However, these techniques have not been investigated for temporal graph extraction.
This paper focuses on the problem of generation of an event-level temporal graph for each document, and we refer to this task as contextualized graph generation. We address this open challenge by proposing a novel reformulation of the task as a sequence-to-sequence mapping problem (Sutskever et al., 2014), which enables us to leverage large pretrained models for our task. Further, different from existing methods, our proposed approach is completely end-to-end and eliminates the need for a pipeline of sub-systems commonly used by traditional methods.
We also address a related open challenge, which is a prerequisite to our main goal: the difficulty of obtaining a large quantity of training graphs with
$^{1}$ Code and pre-trained models available at https://github.com/madaan/temporal-graph-gen
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 864-881 June 6-11, 2021. ©2021 Association for Computational Linguistics
Figure 1: Task overview: given a document (left), automatically extract a temporal graph (right).
human-annotated events and temporal relations. To this end, we automatically produce a large collection of document-graph pairs by using CAEVO, followed by a few rule-based post-processing steps for pruning and noise reduction. We then encode the graph in each training pair as a string in the graph representation format DOT, transforming the text-to-graph mapping into sequence-to-sequence mapping. We fine-tune GPT-2 on this dataset of document-graph pairs, which yields large performance gains over strong baselines on system generated test set and outperforms CAEVO on TimeBank-Dense (Cassidy et al., 2014) on multiple metrics. Figure 1 shows an example of the input document and the generated graph by our system. In summary, our main contributions are:
1. We present the first investigation on using large pre-trained language models for contextualized temporal event graph generation by proposing a new formulation of the problem as a sequence-to-sequence mapping task.
2. We address the difficulty of obtaining a large collection of human-annotated graphs, which is crucial for effective fine-tuning of pretrained models, by automatically producing a collection of 89,000 document-graph pairs.
3. Our experimental results on both the system-generated test set (which allows us to compare the relative performance of different
models) and a hand-labeled, out-of-domain dataset (TimeBank-Dense), show the advantage of our proposed approach over strong baselines. Further, we show that our approach can help in generating plausible answers for open ended-temporal questions in a reading comprehension dataset, Torque (Ning et al., 2020).
Temporal Graph Extraction Tempeval-3 (Uz-Zaman et al., 2013) introduced the task of temporal graph extraction as "the ultimate task for evaluating an end-to-end system that goes from raw text to TimeML annotation". Notable systems developed in response include CAEVO (Chambers et al., 2014), followed by the more recent Cogcomptime (Ning et al., 2018). Both CAEVO and Cogcomptime use several statistical and rule-based methods like event extractors, dependency parsers, semantic role labelers, and time expression identifiers for the task. Our work differs from these systems in both the methodology and desired result in the following ways: i) Instead of using specialized sub-systems, we transform the task into a sequence-to-sequence mapping problem and use a single language model to generate such temporal graphs in an end-to-end fashion from text, subsuming all the intermediate-steps. ii) We develop our system using a corpus of 89,000 documents, which is  $\sim 300x$  larger com
pared to datasets used by CAEVO (36 documents) and Cogcomptime on (276 documents); iii) We remove the noisy events included by CAEVO, but do not limit the extracted events to any specific semantic axis as done by Cogcomptime; and finally, iv) Our method generates graphs where the nodes are not simple verbs but augmented event phrases, containing the subject and the object of each verb. We use CAEVO over Cogcomptime to generate a large-scale corpus for our task and to evaluate our system for the following reasons: i) We found CAEVO to be much more scalable, a critical feature for our task of annotating close to 100k documents, ii) CAEVO over-generates (and not excludes) verbs from its output, giving us the flexibility to filter out noisy events without inadvertently missing out on any critical events. However, our method makes no assumption specific to CAEVO and is adaptable to any other similar system (including Cogcomptime).
Temporal relation extraction We note that the problem of temporal graph extraction is different from the more popular task of Temporal relation extraction (Temprel), which deals with classifying the temporal link between two already extracted events. State of the art Temprel systems use neural methods (Ballesteros et al., 2020; Ning et al., 2019b; Goyal and Durrett, 2019; Han et al., 2019; Cheng and Miyao, 2017), but typically use a handful of documents for their development and evaluation. Vashishtha et al. (2019) are a notable exception by using Amazon Mechanical Turks to obtain manual annotations over a larger dataset of 16,000 sentences. We believe that the techniques presented in our work can be applied to scale the corpus used for training Temprel systems.
Language Models for Graph Generation Recently, Bosselut et al. (2019) proposed COMET, a system that fine-tunes GPT (Radford et al., 2018) on commonsense knowledge graphs like ATOMIC (Sap et al., 2019) and conceptnet (Speer et al., 2017) for commonsense kb completion. Similar to COMET, we adopt large-scale language models for such a conditional generation of text. However, our task differs from COMET in the complexity of both the conditioning text and generated text: we seek to generate temporal graphs grounded in a document, whereas COMET generates a short event/concept phrase conditioned on a relation and an input event/concept phrase. Madaan et al. (2020) and Rajagopal et al. (2021) aim to generate event influx
ence graphs grounded in a situation. Similar to this work, these methods rely on pre-trained language models to generate informative structures grounded in text. Different from us, these methods break the generation process into a sequence of natural language queries. Each query results in an event node, which are finally assembled into a tree. In contrast, we propose a method to directly generate graphs with arbitrary topology from text. Additionally, the events generated by these methods are not present in text making event event prediction, rather than event extraction as their primary focus. You et al. (2018) formulate graphs as a sequence for learning generative models of synthetic and real-world graphs. Similar to their work, we formulate graph generation as an auto-regressive task. However, our goal is the conditional generation of temporal graphs, and not learning unconditional generative distributions. Finally, inspired by recent trends (Raffel et al., 2019), we do not make any graph specific modifications to the model or the decoding process and formulate the problem as a straightforward sequence-to-sequence mapping task. While our approach does not rely on any particular language model, it would be interesting to see the gains achieved by the much larger GPT-3 (Brown et al., 2020) on the dataset produced by our method.<sup>2</sup>
3 Deriving Large-scale Dataset for the Temporal Graph Generation
Definitions and Notations: Let  $\mathbf{G}(\mathbf{V},\mathbf{E})$  be a temporal graph associated with a document  $\mathbf{D}$ , such that vertices  $\mathbf{V}$  are the events in document  $\mathbf{D}$ , and the edges  $\mathbf{E}$  are temporal relations (links) between the events. Every temporal link in  $\mathbf{E}$  takes the form  $r(e_q,e_t)$  where the query event  $e_q$  and the target event  $e_t$  are in  $\mathbf{V}$ , and  $r$  is a temporal relation (e.g., before or after). In this work, we undertake two related tasks of increasing complexity: i) Node generation, and ii) Temporal graph generation:
Task 1: Node Generation: Let  $r(e_q, e_t)$  be an edge in  $\mathbf{E}$ . Let  $C_r$  be the set of sentences in the document  $\mathbf{D}$  that contains the events  $e_q$  or  $e_t$  or are adjacent to them. Given a query consisting of  $C_r$ ,  $r$ , and  $e_q$ , generate  $e_t$ .
Task 2: Temporal Graph Generation: Given a document  $\mathbf{D}$ , generate the corresponding temporal graph  $\mathbf{G}(\mathbf{E},\mathbf{V})$ .
2Not available for research as of April 2021.
Figure 1 illustrates the two tasks. Task 1 is similar to knowledge base completion, except that the output events  $e_{q}$  are generated, and not drawn from a fixed set of events. Task 2 is significantly more challenging, requiring the generation of both the structure and semantics of  $\mathbf{G}$ .
The training data for both the tasks consists of tuples  $\{(x_i, y_i)\}_{i=1}^N$ . For Task 1,  $x_i$  is the concatenation of the query tokens  $(C_r, e_q, r)$ , and  $y_i$  consists of tokens of event  $e_t$ . For Task 2,  $x_i$  is the  $i^{th}$  document  $\mathbf{D_i}$ , and  $y_i$  is the corresponding temporal graph  $\mathbf{G_i}$ .
We use the New York Times (NYT) Annotated Corpus<sup>3</sup> to derive our dataset of document-graph pairs. The corpus has 1.8 million articles written and published by NYT between 1987 and 2007. Each article is annotated with a hand-assigned list of descriptive terms capturing its subject(s). We filter articles with one of the following descriptors: {"bomb", "terrorism", "murder", "riots", "hijacking", "assassination", "kidnapping", "arson", "vandalism", "hate crime", "serial murder", "manslaughter", "extortion"}, yielding 89,597 articles, with a total of 2.6 million sentences and 66 million tokens. For each document D, we use CAEVO (Chambers et al., 2014) to extract the dense temporal graph consisting of i) the set of verbs, and ii) the set of temporal relations between the extracted verbs. CAEVO extracts six temporal relations: before, after, includes, is included, simultaneous, and vague.
We process each dense graph extracted by CAEVO with a series of pruning and augmentation operations: i) We observed that some of the most frequent verbs extracted by CAEVO were the so-called reporting verbs (Liu et al., 2018), like said, say, and told, which do not contribute to the underlying events. For example, said formed nearly  $10\%$  of all the verbs extracted by CAEVO as an event. To remove such noisy events, we remove the five verbs with the lowest inverse document frequencies, as well as an additional set of light and reporting verbs (Liu et al., 2018; Recasens et al., 2010) $^4$  ii) To make event annotations richer, we follow (Chambers and Jurafsky, 2008), and prefix and suffix every verb with its
noun-phrase and object, respectively. This augmentation helps in adding a context to each verb, thus making events less ambiguous. For instance, given a sentence: A called B, after which B called C, CAEVO extracts AFTER(called, called). With the proposed augmentation, the relation becomes AFTER(A called B, B called C), clearly differentiating the two different called events. Our notion of events refers to such augmented verbs. Crucially, different from prior work, our system is trained to extract these augmented event phrases. We also drop all the verbs that do not have either a subject or an object. iii) We remove the relations extracted by the statistical sieves if they have a confidence score of less than 0.50 and retain the rule-based relations as those were shown to be extracted with a high precision by Chambers et al. (2014). Finally, we only retain event-event relations (dropping links between verbs and time expressions) and drop the vague relations as they typically do not play any role in improving the understanding of the temporal sequences in a document. As Table 1 shows, pruning noisy verbs and relations yields sparser and more informative graphs.
Initial	Pruned	% Reduction
#Relations	27,692,365	4,469,298	83.86
#Events	6,733,396	2,615,296	61.15
Table 1: Effect of pruning operations on the number of relations and events.
Creating Sub-graphs using Event Communities We discovered that the (pruned) graph generated for a given document typically has several subgraphs that are either completely disconnected or have high intra-link density. Further, we found that each of these sub-graphs are grounded in different parts of the document. We exploit this phenomenon to map each sub-graph to its correct context, thus reducing the noise in the data.
Relying merely on connectivity for creating subgraphs is still prone to noise, as largely unrelated sub-graphs are often connected via a single event. Instead, we propose a novel approach based on the detection of event communities to divide a graph into sub-graphs, such that the events in a sub-graph are more densely connected to each other. We learn these event communities using the concept of modularity, first introduced by (Newman and Girvan, 2004). We defer the derivation of modularity opti
3https://catalog.ldc.upenn.edu/LDC2008T19
4The final list of verbs is: i) lowidf: "said", "say", "had", "made", "told", ii) light: "appear", "be", "become", "do", "have", "seem", "get", "give", "go", "have", "keep", "make", "put", "set", "take", iii) reporting: "argue", "claim", "say", "suggest", "tell".
mization to the Appendix.
Datasets for Task 1 and Task 2 After running the pruning and clustering operations outlined above on 89k documents, we obtain a corpus of over 890,677 text-graph pairs, with an average of 120.31 tokens per document, and 3.33 events and 4.91 edges per graph. These text-graph pairs constitute the training data for Task 2. We derive the data for Task 1 from the original (undivided) 89k graphs (each document-graph pair contributes multiple examples for Task 1). In Task 1 data, nearly  $80\%$  of the queries  $(C_r, e_q, r)$  had a unique answer  $e_t$ , and nearly  $16\%$  of the queries had two different true  $e_t$ . We retain examples with multiple true  $e_t$  in the training data because they help the model learn diverse temporal patterns that connect two events. For fairness, we retain such cases in the test set. Table 2 lists the statistics of the dataset. The splits were created using non-overlapping documents.
Task	train	valid	test
Task 1	4.26	0.54	0.54
Task 2	0.71	0.09	0.09
Table 2: Dataset statistics (counts in million).
3.1 Graph Representation
We use language models to generate each graph as a sequence of tokens conditioned on the document, thus requiring that the graphs are represented as strings. We use DOT language (Gansner et al., 2006) to format each graph as a string. While our method does not rely on any specific graph representation format, we use DOT as it supports a wide variety of graphs and allows augmenting graphs with node, edge, and graph level information. Further, graphs represented in DOT are readily consumed by popular graph libraries like NetworkX (Hagberg et al., 2008), making it possible to use the graphs for several downstream applications. Figure 2 shows an example graph and the corresponding DOT code. The edges are listed in the order in which their constituent nodes appear in the document. This design choice was inspired by our finding that a vast majority of temporal links exist between events that are either in the same or in the adjoining sentence (this phenomenon was also observed by Ning et al. (2019a)). Thus, listing the edges in the order in which they appear in the document adds a simple inductive bias of lo
cality for the auto-regressive attention mechanism, whereby the attention weights slide from left to right as the graph generation proceeds. Additionally, a fixed order makes the problem well defined, as the mapping between a document and a graph becomes deterministic.
Figure 2: Temporal graph and the corresponding DOT representation for the sentence: Roma clashed fiercely with the police, leading to arrests in which Roma activists said excessive force was used.
The training data  $\mathbf{X}$  for both Tasks 1 and 2 comprises of tuples  $\{(x_i, y_i)\}_{i=1}^N$ . For task 1 (node generation),  $x_i$  the concatenation of context, the source, node, and the relation. The target  $y_i$  consists of the tokens of the target event. For task 2 (graph generation),  $x_i$  is a document and  $y_i$  is the corresponding temporal graph represented in DOT. We train a (separate) conditional language model to solve both the tasks. Specifically, given a training corpus of the form  $\{(x_i, y_i)\}$ , we aim to estimate the distribution  $p_\theta(y_i \mid x_i)$ . Given a training example  $(x_i, y_i)$  we set  $u_i = x_i \| y_i^5$ .  $p_\theta(u_i)$  can then be factorized as a sequence of auto-regressive conditional probabilities using the chain rule:  $p_\theta(u_i) = \prod_{k=1}^n p(u_{i,k} | u_{i,k-1})$ , where  $u_{i,k}$  denotes the  $k^{th}$  token of the  $i^{th}$  sequence, and  $u_{i,k-1}$  denotes the sequence of tokens  $\{u_1, u_2, \ldots, u_{k-1}\}$ . Language models are typically trained by minimizing a cross-entropy loss  $-\log p_\theta(u_i)$  over each sequence  $u_i$  in  $\mathbf{X}$ . However, the cross-entropy loss captures the joint distribution  $p_\theta(x_i, y_i)$ , and is not aligned with our goal of learning conditional distribution  $p_\theta(y_i \mid x_i)$ . To
$5\|$  denotes concatenation
Method	Dataset	BLEU	MTR	RG	ACC
Seq2Seq	TG-Gen (-C)	20.20	14.62	31.95	19.68
Seq2Seq	TG-Gen	21.23	16.48	35.54	20.99
GPT-2	TG-Gen (-C)	36.60	25.11	43.07	35.07
GPT-2	TG-Gen	62.53	43.78	69.10	61.35
Seq2Seq	TB-Dense (-C)	11.55	9.23	21.87	10.06
Seq2Seq	TB-Dense	16.68	12.69	27.75	13.97
GPT-2	TB-Dense (-C)	22.35	15.04	27.73	20.81
GPT-2	TB-Dense	52.21	35.69	57.98	47.91
Table 3: Node Generation (task 1) results.
circumvent this, we train our model by masking the loss terms corresponding to the input  $\boldsymbol{x}_i$ , similar to Bosselut et al. (2019). Let  $m_i$  be a mask vector for each sequence  $\boldsymbol{u}_i$ , set to 0 for positions corresponding to  $\boldsymbol{x}_i$ , and 1 otherwise i.e.  $m_{i,j} = 1$  if  $j > |\boldsymbol{x}_i|$ , else 0. We combine the mask vector with our factorization of  $p_\theta(\boldsymbol{u}_i)$  to formulate a masked language modeling loss  $\mathcal{L}$ , which is minimized over the training corpus  $\mathbf{X}$  to estimate the optimal  $\theta$ :
$$
\mathcal {L} (\mathbf {X}) = - \sum_ {i = 1} ^ {| \mathbf {X} |} \sum_ {j = 1} ^ {| x _ {i} | + | y _ {i} |} m _ {i, j} * \log \left(p _ {\theta} (u _ {i, j} | \boldsymbol {u} _ {i, <   j})\right)
$$
Note that the formulation of masked loss is opaque to the underlying architecture, and can be implemented with a simple change to the loss function. In practice, we use GPT-2 (Radford et al., 2019) based on transformer architecture (Vaswani et al., 2017) for our implementation. Having trained a  $p_{\theta}$  for each task, we generate a node  $(\pmb{y})$  given a query  $(\pmb{x})$  (for Task 1), or a graph  $(\pmb{y})$  given a document  $(\pmb{x})$  (for Task 2) by drawing samples from the appropriate  $p_{\theta}(\pmb{y} \mid \pmb{x})$  using nucleus sampling (Holtzman et al., 2019). We provide more details of our training procedure and the architecture in the Appendix (C.1).
We evaluate our method on two different datasets: i) TG-Gen: Test split of synthetically created dataset (Section 3), and ii) TB-Dense: A mixed-domain corpus, with human-annotated temporal annotations. We create TB-Dense from the test splits of TimeBank-Dense (Cassidy et al., 2014) by applying the same pre-processing operations as we did for TG-Gen. TB-Dense forms a very challenging
dataset for our task because of domain mismatch; our system was trained on a corpus of terrorism-related events, whereas TB-Dense includes documents from a wide array of domains, forming a zero-shot evaluation scenario for our method.
5.2 Implementation Setup
GPT-2: We use GPT-2 medium (355M parameters) for our experiments with 24-layers, a hidden size of 1024, and 16 self-attention heads. We build on the implementation by Wolf et al. (2019), using the default hyperparameters and a block size input sequence length after tokenization) of 512. For optimization, we use AdamW (Loshchilov and Hutter, 2017) with a learning rate of 5e-5, a batch size of 1, and no learning rate scheduling. We also experimented with a block size of 300 and a batch size of 2. We found the results (presented in the appendix) to be worse, underscoring the importance of using a larger block size for generating larger outputs. We generate samples using nucleus sampling using  $p = 0.9$ , and set maximum output length to be 500 for graph generation and 60 foe node generation. All of our experiments were done on a single Nvidia GeForce RTX 2080 Ti. The models were initialized with the pre-trained weights provided by Radford et al. (2019), and fine-tuned for five epochs, with a run-time of 48 hours/epoch for Task 1 and 52 hours/epoch for Task 2. We use the last checkpoint (i.e., at the end of fine-tuning) for all experiments. Despite the higher perplexity on the development set, we found the overall performance of the last checkpoint to be better.