Venue: NAACL 2021

# Neural Language Modeling for Contextualized Temporal Graph Generation

Aman Madaan, Yiming Yang

Language Technologies Institute, Carnegie Mellon University

Pittsburgh, PA, USA

amadanan@cs.cmu.edu

###### Abstract

This paper presents the first study on using large-scale pre-trained language models for automated generation of an event-level temporal graph for a document. Despite the huge success of neural pre-training methods in NLP tasks, its potential for temporal reasoning over event graphs has not been sufficiently explored. Part of the reason is the difficulty in obtaining large training corpora with human-annotated events and temporal links. We address this challenge by using existing IE/NLP tools to automatically generate a large quantity (89,000) of system-produced document-graph pairs, and propose a novel formulation of the contextualized graph generation problem as a sequence-to-sequence mapping task. These strategies enable us to leverage and fine-tune pre-trained language models on the system-induced training data for the graph generation task. Our experiments show that our approach is highly effective in generating structurally and semantically valid graphs. Further, evaluation on a challenging hand-labeled, out-of-domain corpus shows that our method outperforms the closest existing method by a large margin on several metrics. We also show a downstream application of our approach by adapting it to answer open-ended temporal questions in a reading comprehension setting.1

Footnote 1: Code and pre-trained models available at [https://github.com/madaan/temporal-graph-gen](https://github.com/madaan/temporal-graph-gen)

_Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 864–881

June 6-11, 2021. ©2021 Association for Computational Linguistics

## 1 Introduction

Temporal reasoning is crucial for analyzing the interactions among complex events and producing coherent interpretations of text data Duran et al. (2007). There is a rich body of research on the use of temporal information in a variety of important application domains, including topic detection and tracking Makkonen et al. (2003), information extraction Ling and Weld (2010), parsing of clinical records Lin et al. (2016), discourse analysis Evers-Vermeul et al. (2017), and question answering Ning et al. (2020).

Graphs are a natural choice for representing the temporal ordering among events, where the nodes are the individual events, and the edges capture temporal relationships such as "before", "after" or "simultaneous". Representative work on automated extraction of such graphs from textual documents includes the early work by Chambers and Jurafsky (2009), where the focus is on the construction of event chains from a collection of documents, and the more recent caevoChambers et al. (2014) and Cogcomptime Ning et al. (2018), which extract a graph for each input document instead. These methods focus on rule-based and statistical submodules to extract verb-centered events and the temporal relations among them. As an emerging area of nlp, large scale pre-trained language models have made strides in addressing challenging tasks like commonsense knowledge graph completion Bosselut et al. (2019) and task-oriented dialog generation Budzianowski and Vulic (2019). These systems typically fine-tune large language models on a corpus of a task-specific dataset. However, these techniques have not been investigated for temporal graph extraction.

This paper focuses on the problem of generation of an event-level temporal graph for each document, and we refer to this task as _contextualized_ graph generation. We address this open challenge by proposing a novel reformulation of the task as a sequence-to-sequence mapping problem Sutskever et al. (2014), which enables us to leverage large pretrained models for our task. Further, different from existing methods, our proposed approach is completely end-to-end and eliminates the need for a pipeline of sub-systems commonly used by traditional methods.

We also address a related open challenge, which is a prerequisite to our main goal: the difficulty of obtaining a large quantity of training graphs with

human-annotated events and temporal relations. To this end, we automatically produce a large collection of document-graph pairs by using caevo, followed by a few rule-based post-processing steps for pruning and noise reduction. We then encode the graph in each training pair as a string in the graph representation format dot, transforming the text-to-graph mapping into sequence-to-sequence mapping. We fine-tune gpt-2 on this dataset of document-graph pairs, which yields large performance gains over strong baselines on system generated test set and outperforms caevo on TimeBankDense (Cassidy et al., 2014) on multiple metrics. Figure 1 shows an example of the input document and the generated graph by our system. In summary, our main contributions are:

1. We present the first investigation on using large pre-trained language models for contextualized temporal event graph generation by proposing a new formulation of the problem as a sequence-to-sequence mapping task.
2. We address the difficulty of obtaining a large collection of human-annotated graphs, which is crucial for effective fine-tuning of pretrained models, by automatically producing a collection of 89,000 document-graph pairs.
3. Our experimental results on both the system-generated test set (which allows us to compare the relative performance of different models) and a hand-labeled, out-of-domain dataset (TimeBank-Dense), show the advantage of our proposed approach over strong baselines. Further, we show that our approach can help in generating plausible answers for open ended-temporal questions in a reading comprehension dataset, Torque (Ning et al., 2020).

## 2 Related Work

Temporal Graph ExtractionTempeval-3 (UzZaman et al., 2013) introduced the task of temporal graph extraction as "the ultimate task for evaluating an end-to-end system that goes from raw text to TimeML annotation". Notable systems developed in response include caevo Chambers et al. (2014), followed by the more recent Cogcomptime Ning et al. (2018). Both caevo and Cogcomptime use several statistical and rule-based methods like event extractors, dependency parsers, semantic role labelers, and time expression identifiers for the task. Our work differs from these systems in both the methodology and desired result in the following ways: i) Instead of using specialized sub-systems, we transform the task into a sequence-to-sequence mapping problem and use a single language model to generate such temporal graphs in an end-to-end fashion from text, subsuming all the intermediate-steps. ii) We develop our system using a corpus of 89,000 documents, which is \(\sim\) 300x larger com

Figure 1: Task overview: given a document (left), automatically extract a temporal graph (right).

 

pared to datasets used by caevo (36 documents) and Cogcomptime on (276 documents); iii) We remove the noisy events included by caevo, but do not limit the extracted events to any specific semantic axis as done by Cogcomptime; and finally, iv) Our method generates graphs where the nodes are not simple verbs but augmented event phrases, containing the subject and the object of each verb. We use caevo over Cogcomptime to generate a large-scale corpus for our task and to evaluate our system for the following reasons: i) We found caevo to be much more scaleable, a critical feature for our task of annotating close to 100k documents, ii) caevo over-generates (and not excludes) verbs from its output, giving us the flexibility to filter out noisy events without inadvertently missing out on any critical events. However, our method makes no assumption specific to caevo and is adaptable to any other similar system (including Cogcomptime).

Temporal relation extractionWe note that the problem of temporal graph extraction is different from the more popular task of Temporal relation extraction (Tempel), which deals with classifying the temporal link between two already extracted events. State of the art Tempel systems use neural methods Ballesteros et al. (2020); Ning et al. (2019); Goyal and Durrett (2019); Han et al. (2019); Cheng and Miyao (2017), but typically use a handful of documents for their development and evaluation. Vashishtha et al. (2019) are a notable exception by using Amazon Mechanical Turks to obtain manual annotations over a larger dataset of 16,000 sentences. We believe that the techniques presented in our work can be applied to scale the corpus used for training Tempel systems.

Language Models for Graph GenerationRecently, Bosselut et al. (2019) proposed comet, a system that fine-tunes gptRadford et al. (2018) on commonsense knowledge graphs like atomicSap et al. (2019) and conceptnet Speer et al. (2017) for commonsense kb completion. Similar to comet, we adopt large-scale language models for such a conditional generation of text. However, our task differs from comet in the complexity of both the conditioning text and generated text: we seek to generate temporal graphs grounded in a document, whereas comet generates a short event/concept phrase conditioned on a relation and an input event/concept phrase. Madaan et al. (2020) and Rajagopal et al. (2021) aim to generate event influence graphs grounded in a situation. Similar to this work, these methods rely on pre-trained language models to generate informative structures grounded in text. Different from us, these methods break the generation process into a sequence of natural language queries. Each query results in an event node, which are finally assembled into a tree. In contrast, we propose a method to directly generate graphs with arbitrary topology from text. Additionally, the events generated by these methods are not present in text making event event prediction, rather than event extraction as their primary focus. You et al. (2018) formulate graphs as a sequence for learning generative models of synthetic and real-world graphs. Similar to their work, we formulate graph generation as an auto-regressive task. However, our goal is the conditional generation of temporal graphs, and not learning unconditional generative distributions. Finally, inspired by recent trends Raffel et al. (2019), we do not make any graph specific modifications to the model or the decoding process and formulate the problem as a straightforward sequence-to-sequence mapping task. While our approach does not rely on any particular language model, it would be interesting to see the gains achieved by the much larger gpt-3Brown et al. (2020) on the dataset produced by our method.2

Footnote 2: Not available for research as of April 2021.

## 3 Deriving Large-scale Dataset for the Temporal Graph Generation

Definitions and Notations:Let \({\bf G}({\bf V},{\bf E})\) be a temporal graph associated with a document \({\bf D}\), such that vertices \({\bf V}\) are the events in document \({\bf D}\), and the edges \({\bf E}\) are temporal relations (links) between the events. Every temporal link in \({\bf E}\) takes the form \(r(e_{q},e_{t})\) where the query event \(e_{q}\) and the target event \(e_{t}\) are in \({\bf V}\), and \(r\) is a temporal relation (e.g., before or after). In this work, we undertake two related tasks of increasing complexity: i) Node generation, and ii) Temporal graph generation:

Task 1: Node Generation:_Let \(r(e_{q},e_{t})\) be an edge in \({\bf E}\). Let \(C_{r}\) be the set of sentences in the document \({\bf D}\) that contains the events \(e_{q}\) or \(e_{t}\) or are adjacent to them. Given a query consisting of \(C_{r}\), \(r\), and \(e_{q}\), generate \(e_{t}\)._

Task 2: Temporal Graph Generation:_Given a document \({\bf D}\), generate the corresponding temporal graph \({\bf G}({\bf E},{\bf V})\).

 

Figure 1 illustrates the two tasks. Task 1 is similar to knowledge base completion, except that the output events \(e_{q}\) are generated, and not drawn from a fixed set of events. Task 2 is significantly more challenging, requiring the generation of both the structure and semantics of **G**.

The training data for both the tasks consists of tuples \(\{(x_{i},y_{i})\}_{i=1}^{N}\). For Task 1, \(x_{i}\) is the concatenation of the query tokens \((C_{r},e_{q},r)\), and \(y_{i}\) consists of tokens of event \(e_{t}\). For Task 2, \(x_{i}\) is the \(i^{th}\) document \(\textbf{D_{l}}\), and \(y_{i}\) is the corresponding temporal graph \(\textbf{G_{i}}\).

We use the New York Times (nyt) Annotated Corpus 3 to derive our dataset of document-graph pairs. The corpus has 1.8 million articles written and published by nyt between 1987 and 2007. Each article is annotated with a hand-assigned list of descriptive terms capturing its subject(s). We filter articles with one of the following descriptors: {"bomb", "terrorism", "murder", "riots", "hijacking", "assassination", "kidnapping", "arson", "vandalism", "hate crime", "serial murder", "manslaughter", "extortion"), yielding 89,597 articles, with a total of 2.6 million sentences and 66 million tokens. For each document **D**, we use caevo[4] to extract the dense temporal graph consisting of i) the set of verbs, and ii) the set of temporal relations between the extracted verbs. caevo extracts six temporal relations: before, after, includes, is included, simultaneous, and vague.

Footnote 3: [https://catalog.ldc.upenn.edu/LDC2008T19](https://catalog.ldc.upenn.edu/LDC2008T19)

We process each dense graph extracted by caevo with a series of pruning and augmentation operations: **i)** We observed that some of the most frequent verbs extracted by caevo were the so-called reporting verbs [10], like _said_, _say_, and _ told_, which do not contribute to the underlying events. For example, _said_ formed nearly 10% of all the verbs extracted by caevo as an event. To remove such noisy events, we remove the five verbs with the lowest inverse document frequencies, as well as an additional set of light and reporting verbs [10; 11]4**ii)** To make event annotations richer, we follow [4], and prefix and suffix every verb with its noun-phrase and object, respectively. This augmentation helps in adding a context to each verb, thus making events less ambiguous. For instance, given a sentence: _A called B, after which B called C_, caevo extracts after(\(called,called\)). With the proposed augmentation, the relation becomes after(_A called B, B called C_), clearly differentiating the two different _called_ events. Our notion of events refers to such augmented verbs. Crucially, different from prior work, our system is trained to extract these augmented event phrases. We also drop all the verbs that do not have either a subject or an object. **iii)** We remove the relations extracted by the statistical sieves if they have a confidence score of less than 0.50 and retain the rule-based relations as those were shown to be extracted with a high precision by chambers2014. Finally, we only retain event-event relations (dropping links between verbs and time expressions) and drop the vague relations as they typically do not play any role in improving the understanding of the temporal sequences in a document. As Table 1 shows, pruning noisy verbs and relations yields sparser and more informative graphs.

Footnote 4: The final list of verbs is: i) _low idf_: “said”, “say”, “had”, “made, “told”, ii) _light:_ “appear”, “be”, “become”, “do”, “have”, “seen”, “get”, “give”, “go”, “have”, “keep”, “make”, “put”, “set”, “take”, iii) _reporting:_ “argue”, “claim”, “say”, “suggest”, “ttell”.

Creating Sub-graphs using Event CommunitiesWe discovered that the (pruned) graph generated for a given document typically has several subgraphs that are either completely disconnected or have high intra-link density. Further, we found that each of these sub-graphs are grounded in different parts of the document. We exploit this phenomenon to map each sub-graph to its correct context, thus reducing the noise in the data.

Relying merely on connectivity for creating subgraphs is still prone to noise, as largely unrelated sub-graphs are often connected via a single event. Instead, we propose a novel approach based on the detection of event communities to divide a graph into sub-graphs, such that the events in a sub-graph are more densely connected to each other. We learn these event communities using the concept of modularity, first introduced by [11]. We defer the derivation of modularity opti

\begin{table}
\begin{tabular}{l r r r} \hline \hline  & Initial & Pruned & \% Reduction \\ \hline \#Relations & 27,692,365 & 4,469,298 & 83.86 \\ \#Events & 6,733,396 & 2,615,296 & 61.15 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Effect of pruning operations on the number of relations and events.

 

mization to the Appendix.

**Datasets for Task 1 and Task 2** After running the pruning and clustering operations outlined above on 89k documents, we obtain a corpus of over 890,677 text-graph pairs, with an average of 120.31 tokens per document, and 3.33 events and 4.91 edges per graph. These text-graph pairs constitute the training data for Task 2. We derive the data for Task 1 from the original (undivided) 89k graphs (each document-graph pair contributes multiple examples for Task 1). In Task 1 data, nearly 80% of the queries \((C_{r},e_{q},r)\) had a unique answer \(e_{t}\), and nearly 16% of the queries had two different true \(e_{t}\). We retain examples with multiple true \(e_{t}\) in the training data because they help the model learn diverse temporal patterns that connect two events. For fairness, we retain such cases in the test set. Table 2 lists the statistics of the dataset. The splits were created using non-overlapping documents.

### Graph Representation

We use language models to generate each graph as a sequence of tokens conditioned on the document, thus requiring that the graphs are represented as strings. We use dot language Gansner et al. (2006) to format each graph as a string. While our method does not rely on any specific graph representation format, we use dot as it supports a wide variety of graphs and allows augmenting graphs with node, edge, and graph level information. Further, graphs represented in dot are readily consumed by popular graph libraries like NetworkX Hagberg et al. (2008), making it possible to use the graphs for several downstream applications. Figure 2 shows an example graph and the corresponding dot code. The edges are listed in the order in which their constituent nodes appear in the document. This design choice was inspired by our finding that a vast majority of temporal links exist between events that are either in the same or in the adjoining sentence (this phenomenon was also observed by Ning et al. (2019a)). Thus, listing the edges in the order in which they appear in the document adds a simple inductive bias of locality for the auto-regressive attention mechanism, whereby the attention weights _slide_ from left to right as the graph generation proceeds. Additionally, a fixed order makes the problem well defined, as the mapping between a document and a graph becomes deterministic.

## 4 Model

The training data \(\mathbf{X}\) for both Tasks 1 and 2 comprises of tuples \(\{(\mathbf{x_{i}},\mathbf{y_{i}})\}_{i=1}^{N}\). For task 1 (node generation), \(\mathbf{x_{i}}\) the concatenation of context, the source, node, and the relation. The target \(\mathbf{y_{i}}\) consists of the tokens of the target event. For task 2 (graph generation), \(\mathbf{x_{i}}\) is a document and \(\mathbf{y_{i}}\) is the corresponding temporal graph represented in dot. We train a (separate) conditional language model to solve both the tasks. Specifically, given a training corpus of the form \(\{(\mathbf{x_{i}},\mathbf{y_{i}})\}\), we aim to estimate the distribution \(p_{\theta}(\mathbf{y_{i}}\mid\mathbf{x_{i}})\). Given a training example \((\mathbf{x_{i}},\mathbf{y_{i}})\) we set \(\mathbf{u_{i}}=\mathbf{x_{i}}\|\mathbf{y_{i}}\lx@note{footnote}{$\parallel$ denotes concatenation}\). \(p_{\theta}(\mathbf{u_{i}})\) can then be factorized as a sequence of auto-regressive conditional probabilities using the chain rule: \(p_{\theta}(\mathbf{u_{i}})=\prod_{k=1}^{n}p(u_{i,k}|\mathbf{u_{i}<k})\), where \(u_{i,k}\) denotes the \(k^{th}\) token of the \(i^{th}\) sequence, and \(\mathbf{u_{i},<k}\) denotes the sequence of tokens \(\{u_{1},u_{2},...,u_{k-1}\}\). Language models are typically trained by minimizing a cross-entropy loss \(-\text{log}p_{\theta}(\mathbf{u_{i}})\) over each sequence \(\mathbf{u_{i}}\) in \(\mathbf{X}\). However, the cross-entropy loss captures the joint distribution \(p_{\theta}(\mathbf{x_{i}},\mathbf{y_{i}})\), and is not aligned with our goal of learning conditional distribution \(p_{\theta}(\mathbf{y_{i}}\mid\mathbf{x_{i}})\). To

Figure 2: Temporal graph and the corresponding dot representation for the sentence: _Roma clashed fiercely with the police, leading to arrests in which Roma activists said excessive force was used._



circumvent this, we train our model by masking the loss terms corresponding to the input \({\mathbf{x_{i}}}\), similar to Bosselut et al. (2019). Let \({\mathbf{m_{i}}}\) be a mask vector for each sequence \({\mathbf{u_{i}}}\), set to \(0\) for positions corresponding to \({\mathbf{x_{i}}}\), and \(1\) otherwise i.e. \(m_{i,j}=1\) if \(j>|{\mathbf{x_{i}}}|\), else 0. We combine the mask vector with our factorization of \(p_{\theta}\left({\mathbf{u_{i}}}\right)\) to formulate a _masked_ language modeling loss \(\mathcal{L}\), which is minimized over the training corpus \(\mathbf{X}\) to estimate the optimal \(\theta\):

\[\mathcal{L}(\mathbf{X})=-\sum_{i=1}^{|\mathbf{X}|}\sum_{j=1}^{|x_{i}|+|y_{i} |}m_{i,j}*\log\left(p_{\theta}\left(u_{i,j}|{\mathbf{u_{i}}},<{\mathbf{z}}\right)\right)\]

Note that the formulation of masked loss is opaque to the underlying architecture, and can be implemented with a simple change to the loss function. In practice, we use gpt-2Radford et al. (2019) based on transformer architecture Vaswani et al. (2017) for our implementation. Having trained a \(p_{\theta}\) for each task, we generate a node (\({\mathbf{y}}\)) given a query (\({\mathbf{x}}\)) (for Task 1), or a graph (\({\mathbf{y}}\)) given a document (\({\mathbf{x}}\)) (for Task 2) by drawing samples from the appropriate \(p_{\theta}({\mathbf{y}}\mid{\mathbf{x}})\) using nucleus sampling Holtzman et al. (2019). We provide more details of our training procedure and the architecture in the Appendix (C.1).

## 5 Experiments and Results

### Evaluation Datasets

We evaluate our method on two different datasets:

i) **TG-Gen**: Test split of synthetically created dataset (Section 3), and ii) **TB-Dense:** A mixed-domain corpus, with human-annotated temporal annotations. We create TB-Dense from the test splits of TimeBank-Dense Cassidy et al. (2014) by applying the same pre-processing operations as we did for TG-Gen. TB-Dense forms a very challenging dataset for our task because of domain mismatch; our system was trained on a corpus of terrorism-related events, whereas TB-Dense includes documents from a wide array of domains, forming a zero-shot evaluation scenario for our method.

### Implementation Setup

gpt-2:We use gpt-2 medium (355M parameters) for our experiments with 24-layers, a hidden size of 1024, and 16 self-attention heads. We build on the implementation by Wolf et al. (2019), using the default hyperparameters and a block size input sequence length after tokenization) of 512. For optimization, we use AdamW Loshchilov and Hutter (2017) with a learning rate of 5e-5, a batch size of 1, and no learning rate scheduling. We also experimented with a block size of 300 and a batch size of 2. We found the results (presented in the appendix) to be worse, underscoring the importance of using a larger block size for generating larger outputs. We generate samples using nucleus sampling using \(p=0.9\), and set maximum output length to be 500 for graph generation and 60 foe node generation. All of our experiments were done on a single Nvidia GeForce rtx 2080 Ti. The models were initialized with the pre-trained weights provided by Radford et al. (2019), and fine-tuned for five epochs, with a run-time of 48 hours/epoch for Task 1 and 52 hours/epoch for Task 2. We use the last checkpoint (i.e., at the end of fine-tuning) for all experiments. Despite the higher perplexity on the development set, we found the overall performance of the last checkpoint to be better.

We also experimented with gpt-2 without fine-tuning (i.e., by directly using pre-trained weights). The non-finetuned GPT-2 fared poorly for both the tasks across all the metrics, getting a bleu score of near 0 for Task 1. This dismal performance underscores the importance of fine-tuning on the end task for large-scale pre-trained language models.

Finally, we note that our method does not make any model-specific assumption, and can be used with any auto-regressive language model (i.e., a language model that generates a sequence of tokens from left-to-right). We use gpt-2 as a representative for large pre-trained language models.

Seq2Seq:We train a bi-directional lstmHochreiter and Schmidhuber (1997) based sequence-to-sequence model Bahdanau et al. (2015) with global attention Luong et al. (2015) and a hidden size of 500 as a baseline to contrast with gpt-2. The to

\begin{table}
\begin{tabular}{l c r r r r} \hline Method & Dataset & BLEU & mtr & rg & acc \\ \hline Seq2Seq & tG-Gen (-C) & 20.20 & 14.62 & 31.95 & 19.68 \\ Seq2Seq & tG-Gen & 21.23 & 16.48 & 35.54 & 20.99 \\ GPT-2 & tG-Gen (-C) & 36.60 & 25.11 & 43.07 & 35.07 \\ GPT-2 & tG-Gen & **62.53** & **43.78** & **69.10** & **61.35** \\ \hline Seq2Seq & tB-Dense (-C) & 11.55 & 9.23 & 21.87 & 10.06 \\ Seq2Seq & tB-Dense & 16.68 & 12.69 & 27.75 & 13.97 \\ GPT-2 & tB-Dense (-C) & 22.35 & 15.04 & 27.73 & 20.81 \\ GPT-2 & tB-Dense & **52.21** & **35.69** & **57.98** & **47.91** \\ \hline \end{tabular}
\end{table}
Table 3: Node Generation (task 1) results.

 

ken embeddings initialized using 300-dimensional pre-trained Glove Pennington et al. (2014).

### Task 1: Node Generation

**Paragraph: Mr. Grier, a former defensive lineman for the New York Giants who was ordained as a minister in 1986, testified on Dec. 9 that he had visited Mr. Simpson a month earlier**

MetricsGiven a query \((C_{r},e_{q},r)\), with \(C_{r}\) being the context (sentences containing events \(e_{q},e_{t}\) and their neighboring sentences) and \(e_{q}\) as the source event, Task 1 is to generate a target event \(e_{t}\) such that \(r(e_{q},e_{t})\). We format each query as "In the context of \(C\), what happens \(r~{}e_{q}\)?". We found formatting the query in natural language to be empirically better. Let \(\hat{e_{t}}\) be the system generated event. We compare \(e_{t}\) vs. \(\hat{e_{t}}\) using bleuPapineni et al. (2002), meteorDenkowski and Lavie (2011), and rougeLin (2004)6, and measure the accuracy (acc) as the fraction of examples where \(e_{t}=\hat{e_{t}}\).

Footnote 6: Sharma et al. (2017), [https://github.com/Maluuba/nlg-eval](https://github.com/Maluuba/nlg-eval)

**Results on TG-Gen** The results are listed in Table 3. Unsurprisingly, gpt-2 achieves high scores across the metrics showing that it is highly effective in generating correct events. To test the generative capabilities of the models, we perform an ablation by removing the sentence containing the target event \(e_{t}\) from \(C_{r}\) (indicated with -\(C\)). Removal of context causes a drop in performance for both gpt-2 and Seq2Seq, showing that it is crucial for generating temporal events. However, gpt-2 obtains higher relative gains with context present, indicating that it uses its large architecture and pre-training to use the context more efficiently. gpt-2 also fares better as compared with Seq2Seq in terms of drop in performance for the out-of-domain TB-Dense dataset on metrics like accuracy (\(-21\%\) vs. \(-33\%\)) and bleu (\(-16\%\) vs. \(-21\%\)), indicating that pre-training makes helps gpt-2 in generalizing across the domains.

**Human Evaluation** To understand the nature of errors, we analyzed 100 randomly sampled incorrect generations. For 53% of the errors, gpt-2 generated a non-salient event which nevertheless had the correct temporal relation with the query. Interestingly, for 10% of the events, we found that gpt-2 _fixed_ the label assigned by caevoTable 4, i.e., \(e_{t}\) was incorrect but \(\hat{e_{t}}\) was correct.

### Task 2: Graph Generation

**MetricsLet \(\mathbf{G_{i}(V_{i},E_{i})}\) and \(\mathbf{\hat{G_{i}}(\hat{V_{i}},\hat{E_{i}})}\) be the true and the generated graphs for an example \(i\) in the test corpus. Please recall that our proposed method generates a graph from a given document as a string in dot. Let \(\mathbf{y_{i}}\) and \(\hat{\mathbf{y_{i}}}\) be the string representations of the true and generated graphs. We evaluate our generated graphs using three types of metrics:**

1. **Graph string metrics**: To compare \(\mathbf{y_{i}}\) vs. \(\hat{\mathbf{y_{i}}}\), we use bleu, meteor, and rouge, and also measure parse accuracy (dot%) as the % of generated graphs \(\hat{\mathbf{y_{i}}}\) which are valid dot files.

2. **Graph structure metrics** To compare the structures of the graphs \(\mathbf{G_{i}}\) vs. \(\mathbf{\hat{G_{i}}}\), we use i) Graph edit distance (ged) Abu-Aisheh et al. (2015) - the minimum numbers of edits required to transform the predicted graph to the true graph by addition/removal of an edge/node; ii) Graph isomorphism (iso) Cordella et al. (2001) - a binary measure set to \(1\) if the graphs are isomorphic (without considering the node or edge attributes); iii) The average graph size \((|\mathbf{V_{i}}|,|\mathbf{E_{i}}|,|\hat{\mathbf{V_{i}}}|,|\mathbf{\hat{E_{i}} }|)\) and the average degree (d\((\text{V})\)).

3. **Graph semantic metrics:** We evaluate the node sets (\(\mathbf{V_{i}}\) vs. \(\hat{\mathbf{V_{i}}}\)) and the edge sets (\(\mathbf{E_{i}}\) vs. \(\hat{\mathbf{E_{i}}}\)) to compare the semantics of the true and generated

\begin{table}
\begin{tabular}{l l r r r r r} \hline  & Dataset & \multicolumn{1}{c}{\(v_{P}\)} & \multicolumn{1}{c}{\(v_{R}\)} & \multicolumn{1}{c}{\(v_{F_{i}}\)} & \multicolumn{1}{c}{\(e_{P}\)} & \multicolumn{1}{c}{\(e_{R}\)} & \multicolumn{1}{c}{\(e_{f_{i}}\)} \\ \hline Seq2Seq & TG-Gen & 36.84 & 24.89 & 28.11 & 9.65 & 4.29 & 4.70 \\ GPT-2 & TG-Gen & **69.31** & **66.12** & **66.34** & **27.95** & **25.89** & **25.22** \\ \hline Seq2Seq & TB-Dense & 24.86 & 15.25 & 17.99 & 4.7 & 0.14 & 0.24 \\ CAEVO & TB-Dense & 37.53 & **79.83** & **48.96** & 7.95 & **14.62** & **8.96** \\ GPT-2 & TB-Dense & **45.96** & 48.44 & 44.97 & **8.74** & 8.89 & 7.96 \\ \hline \end{tabular}
\end{table}
Table 6: Graph semantic metrics.

\begin{table}
\begin{tabular}{l c c c c c} \hline  & Dataset & bleu & mtr & rg & dot\% \\ \hline Seq2Seq & TG-Gen & 4.79 & 15.03 & 45.95 & 86.93 \\ gpt-2 & TG-Gen & **37.77** & **37.22** & **64.24** & **94.47** \\ \hline Seq2Seq & TB-Dense & 2.61 & 12.76 & 28.36 & 89.31 \\ gpt-2 & TB-Dense & **26.61** & **29.49** & **49.26** & **92.37** \\ \hline \end{tabular}
\end{table}
Table 4: An example of gpt-2 _fixing_ the label given by caevo. Given a query _event after “Mr. Grier visited”_, caevo incorrectly extracts _Mr. Grier odained_, whereas gpt-2 generates the correct event: _Mr. Grier testified_.

\begin{table}
\begin{tabular}{l l r r r r r} \hline  & Dataset & \multicolumn{1}{c}{\(v_{P}\)} & \multicolumn{1}{c}{\(v_{R}\)} & \multicolumn{1}{c}{\(v_{F_{i}}\)} & \multicolumn{1}{c}{\(e_{P}\)} & \multicolumn{1}{c}{\(e_{R}\)} & \multicolumn{1}{c}{\(e_{F_{i}}\)} \\ \hline Seq2Seq & TG-Gen & 36.84 & 24.89 & 28.11 & 9.65 & 4.29 & 4.70 \\ GPT-2 & TG-Gen & **69.31** & **66.12** & **66.34** & **27.95** & **25.89** & **25.22** \\ \hline Seq2Seq & TB-Dense & 24.86 & 15.25 & 17.99 & 4.7 & 0.14 & 0.24 \\ CAEVO & TB-Dense & 37.53 & **79.83** & **48.96** & 7.95 & **14.62** & **8.96** \\ GPT-2 & TB-Dense & **45.96** & 48.44 & 44.97 & **8.74** & 8.89 & 7.96 \\ \hline \end{tabular}
\end{table}
Table 5: Graph string metrics.

 

graphs. For every example \(i\), we calculate node-set precision, recall, and \(F_{1}\) score, and average them over the test set to obtain node precision (\(v_{P}\)), recall (\(v_{R}\)), and \(F_{1}\) (\(v_{F}\)). We evaluate the predicted edge set using temporal awareness UzZaman and Allen (2012); UzZaman et al. (2013). For an example \(i\), we calculate \(e^{i}_{P}=\frac{|\mathbf{E}^{*}_{i}\cap\mathbf{E}^{*}_{i}|}{|\mathbf{E}^{*}_{ i}|},e^{i}_{R}=\frac{|\mathbf{E}^{*}_{i}\cap\mathbf{E}_{i}|^{-}|}{|\mathbf{ E}_{i}|^{-}|}\) where symbol \(+\) denotes the temporal transitive closure Allen (1983) of the edge set. Similarly, \(-\) indicates the reduced edge set, obtained by removing all the edges that can be inferred from other edges transitively. The \(F_{1}\) score \(e^{i}_{F_{1}}\) is the harmonic mean of \(e^{i}_{P}\) and \(e^{i}_{R}\), and these metrics are averaged over the test set to obtain the temporal awareness precision (\(e_{P}\)), recall (\(e_{R}\)), and \(F_{1}\) score (\(e_{F_{1}}\)). Intuitively, the node metrics judge the quality of generated events in the graph, and the edge metrics evaluate the corresponding temporal relations.

ResultsTables 5, 7, and 6 present results for graph generation, and we discuss them next.

gpt-2 vs. Seq2Seq gpt-2 outperforms Seq2Seq on all the metrics by a large margin in both fine-tuned (TG-Gen) and zero-shot settings (TB-Dense). gpt-2 generated graphs are closer to the true graphs in size and topology, as shown by lower edit distance and a higher rate of isomorphism in Table 7. Both the systems achieve high parsing rates (dot %), with gpt-2 generating valid dot files 94.6% of the time. The high parsing rates are expected, as even simpler architectures like vanilla RNNs have been shown to generate syntactically valid complex structures like LaTeXdocuments with ease Karpathy (2015).

gpt-2 vs. caevoWe compare the graphs generated by gpt-2 with those extracted by caevo Chambers et al. (2014)7 from the TB-Dense documents. We remove all the vague edges and the light verbs from the output of caevo for a fair comparison. Please recall that caevo is the tool we used for creating the training data for our method. Further, caevo was trained using TB-Dense, while gpt-2 was not. Thus, caevo forms an upper bound over the performance of gpt-2. The results in Tables 6 and 7 show that despite these challenges, gpt-2 performs strongly across a wide range of metrics, including ed, iso, and temporal awareness. Comparing the node-set metrics, we see that gpt-2 leads caevo by over eight precision points (\(v_{P}\)), but loses on recall (\(v_{R}\)) as caevo extracts nearly every verb in the document as a potential event. On temporal awareness (edge-metrics), gpt-2 outperforms both caevo and Seq2Seq in terms of average precision score \(e_{P}\) and achieves a competitive \(e_{F_{1}}\) score. These results have an important implication: they show that our method can best or match a pipeline of specialized systems given reasonable amounts of training data for temporal graph extraction. caevo involves several sub-modules to perform part-of-speech tagging, dependency parsing, event extraction, and several statistical and rule-based systems for temporal extraction. In contrast, our method involves no hand-curated features, is trained end-to-end (single gpt-2), and can be easily scaled to new datasets.

Footnote 7: [https://github.com/nchambers/caevo](https://github.com/nchambers/caevo)

Node extraction and Edge ExtractionThe node-set metrics in Table 6 shows that gpt-2 avoids generating noisy events (high \(P\)), and extracts salient events (high \(R\)). This is confirmed by manual analysis, done by randomly sampling 100 graphs from the gpt-2 generated graphs and isolating the main verb in each node (Table 8). We provide several examples of generated graphs in the Appendix. We note from Table 6 that the relative difference between the \(e_{F_{1}}\) scores for gpt-2 and Seq2Seq (25.22 vs. 4.70) is larger than the relative difference between their \(v_{F_{1}}\) scores (66.34 vs. 28.11), showing that edge-extraction is the more challenging task which allows gpt-2 to take full advantage of its powerful architecture. We also observe that edge extraction (\(e_{F_{1}}\)) is highly sensitive

\begin{table}
\begin{tabular}{l l r r r r r} \hline \hline  & Dataset & \(|\mathbf{V}|\) & \(|\mathbf{E}|\) & \(d(\mathbf{V})\) & \(\text{ged}\downarrow\) & iso \(\uparrow\) \\ \hline True & TG-Gen & 4.15 & 5.47 & 1.54 & 0 & 100 \\ Seq2Seq & TG-Gen & 2.24 & 2.23 & 1.12 & 6.09 & 32.49 \\ gpt-2 & TG-Gen & **3.81** & **4.60** & **1.40** & **2.62** & **41.66** \\ \hline True & TB-Dense & 4.39 & 6.12 & 2.02 & 0 & 100 \\ Seq2Seq & TB-Dense & 2.21 & 2.20 & 1.11 & 6.22 & 23.08 \\ caevo & TB-Dense & 10.73 & 17.68 & 2.76 & 18.68 & 11.11 \\ gpt-2 & TB-Dense & **3.72** & **4.65** & **1.75** & **4.05** & **24.00** \\ \hline \hline \end{tabular}
\end{table}
Table 7: Graph structure metrics.

 

to node extraction (\(v_{F_{1}}\)); for gpt-2, a 27% drop in \(v_{F_{1}}\) (66.34 on TG-Gen vs. 44.97 on TB-Dense) causes a 68% drop in \(e_{F_{1}}\) (25.22 on TG-Gen vs. 7.96 on TB-Dense). As each node is connected to multiple edges on average (Table 7), missing a node during the generation process might lead to multiple edges being omitted, thus affecting edge extraction metrics disproportionately.

### Answering for Open-ended Questions

A benefit of our approach of using a pre-trained language model is that it can be used to _generate_ an answer for open-ended temporal questions. Recently, Ning et al. (2020) introduced Torque, a temporal reading-comprehension dataset. Several questions in Torque have no answers, as they concern a time scope not covered by the passage (the question is about events not mentioned in the passage). We test the ability of our system for generating plausible answers for such questions out of the box (i.e., without training on Torque). Given a (passage, question) pair, we create a query \((C,e_{q},r)\), where \(C\) is the passage, and \(e_{q}\) and \(r\) are the query event and temporal relation in the question. We then use our gpt-2 based model for node-generation trained without context and generate an answer \(e_{t}\) for the given query. A human-judge rated the answers generated for 100 such questions for plausibility, rating each answer as being _plausible_, _somewhat plausible_, or _incorrect_. For each answer rated as either _plausible_ or _somewhat plausible_, the human-judge wrote a short explanation to provide a rationale for the plausibility of the generated event. Out of the 100 questions, the human-judge rated 22 of the generated answers as plausible and ten as somewhat plausible, showing the promise of our method on this challenging task (Table 9).

## 6 Conclusion and Future Work

Current methods for generating event-level temporal graphs are developed with relatively small amounts of hand-labeled data. On the other hand, the possibility of using pre-trained language models for this task has not received sufficient attention. This paper addresses this open challenge by first developing a data generation pipeline that uses existing iE/NLP/clustering techniques for automated acquisition of a large corpus of document-graph pairs, and by proposing a new formulation of the graph generation task as a sequence-to-sequence mapping task, allowing us to leverage and fine-tune pretrained language models. Our experiments strongly support the effectiveness of the proposed approach, which significantly outperforms strong baselines. We plan to explore techniques for adapting large-scale language models on unseen domains and at multiple granularity levels in the future.

## Acknowledgments

Thanks to Nathanael Chambers and Dheeraj Rajagopal for the helpful discussion, and to the anonymous reviewers for their constructive feedback. This material is based on research sponsored in part by the Air Force Research Laboratory under agreement number FA8750-19-2-0200. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation thereon. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of the Air Force Research Laboratory or the U.S. Government.

## References

* Abu-Asisheh et al. (2015) Zeina Abu-aisheh, Romain Raveaux, Jean-Yves Ramel, and Patrick Martineau. 2015. An exact graph edit distance algorithm for solving pattern recognition problems. In An exact graph edit distance algorithm for solving pattern recognition problems. Cited by: SS1.
* F Allen (1983) Maintaining knowledge about temporal intervals. Communications of the ACM26 (11), pp. 832-843.
* Krizhevsky et al. (2012) Krizhevsky, Ilya Sutskever, and Yoshua Bengio. 2012. arXiv preprint arXiv:1206.03718.
* Krizhevsky et al. (2013) Krizhevsky, Ilya Sutskever, and Yoshua Bengio. 2013. arXiv preprint arXiv:1303.1266.
* Krizhevsky et al. (2014) Krizhevsky, Ilya Sutskever, and Yoshua Bengio. 2014. arXiv preprint arXiv:1406.11002.
* Krizhevsky et al. (2015) Krizhevsky, Ilya Sutskever, and Yoshua Bengio. 2015. arXiv preprint arXiv:1503.03718.
* Krizhevsky et al. (2016) Krizhevsky, Ilya Sutskever, and Yoshua Bengio. 2016. arXiv preprint arXiv:1606.03718.
* Krizhevsky et al. (2017) Krizhevsky, Ilya Sutskever, and Yoshua Bengio. 2017. arXiv preprint arXiv:1706.03718.
* Krizhevsky et al. (2018) Krizhevsky, Ilya Sutskever, and Yoshua Bengio. 2018. arXiv preprint arXiv:1803.03718.
* Krizhevsky et al. (2019) Krizhevsky, Ilya Sutskever, and Yoshua Bengio. 2019. arXiv preprint arXiv:1907.03718.
* Krizhevsky et al. (2019) Krizhevsky, Ilya Sutskever, and Yoshua Bengio. 2019. arXiv preprint arXiv:1907.

 

Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016. Layer normalization. _arXiv preprint arXiv:1607.06450_.
* Bahdanau et al. (2015) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In _3rd International Conference on Learning Representations, ICLR 2015_.
* Ballesteros et al. (2020) Miguel Ballesteros, Rishita Anubhai, Shuai Wang, Nima Pourdamghani, Yogarshi Vyas, Jie Ma, Parminder Bhatia, Kathleen McKeown, and Yaser Al-Onaizan. 2020. Severing the edge between before and after: Neural architectures for temporal ordering of events. _arXiv preprint arXiv:2004.04295_.
* Bosselut et al. (2019) Antoine Bosselut, Hannah Rashkin, Maarten Sap, Chaitanya Malaviya, Asli Celikyilmaz, and Yejin Choi. 2019. Comet: Commonsense transformers for automatic knowledge graph construction. In _ACL_.
* Brown et al. (2020) Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prrulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. _arXiv preprint arXiv:2005.14165_.
* Budzianowski and Vulic (2019) Pawel Budzianowski and Ivan Vulic. 2019. Hello, it's gpt-2-how can i help you? towards the use of pre-trained language models for task-oriented dialogue systems. _arXiv preprint arXiv:1907.05774_.
* Cassidy et al. (2014) Taylor Cassidy, Bill McDowell, Nathaneh Chambers, and Steven Bethard. 2014. An annotation framework for dense event ordering. Technical report, CARNEGIE-MELLON UNIV PITTSBURGH PA.
* Chambers et al. (2014) Nathanael Chambers, Taylor Cassidy, Bill McDowell, and Steven Bethard. 2014. Dense event ordering with a multi-pass architecture. _Transactions of the Association for Computational Linguistics_, 2:273-284.
* Chambers and Jurafsky (2008) Nathanael Chambers and Dan Jurafsky. 2008. Unsupervised learning of narrative event chains. In _Proceedings of ACL-08: HLT_, pages 789-797.
* Chambers and Jurafsky (2009) Nathanael Chambers and Dan Jurafsky. 2009. Unsupervised learning of narrative schemas and their participants. In _Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP_, pages 602-610.
* Cheng and Miyao (2017) Fei Cheng and Yusuke Miyao. 2017. Classifying temporal relations by bidirectional lstm over dependency paths. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_, pages 1-6.
* Clauset et al. (2004) Aaron Clauset, Mark EJ Newman, and Cristopher Moore. 2004. Finding community structure in very large networks. _Physical review E_, 70(6):066111.
* Cordella et al. (2001) Luigi Pietro Cordella, Pasquale Foggia, Carlo Sansone, and Mario Vento. 2001. An improved algorithm for matching large graphs. In _3rd IAPR-TC15 workshop on graph-based representations in pattern recognition_, pages 149-159.
* Denkowski and Lavie (2011) Michael Denkowski and Alon Lavie. 2011. Meteor 1.3: Automatic metric for reliable optimization and evaluation of machine translation systems. In _Proceedings of the sixth workshop on statistical machine translation_, pages 85-91. Association for Computational Linguistics.
* Duran et al. (2007) Nicholas Duran, Philip M McCarthy, Art C Graesser, and Danielle S McNamara. 2007. Using temporal cohesion to predict temporal coherence in narrative and expository texts. _Behavior Research Methods_, 39(2):212-223.
* Evers-Vermeul et al. (2017) Jacqueline Evers-Vermeul, Jet Hoek, and Merel CJ Scholman. 2017. On temporality in discourse annotation: Theoretical and practical considerations. _Dialogue & Discourse_, 8(2):1-20.
* Gansner et al. (2006) Emden Gansner, Eleftherios Koutsofios, and Stephen North. 2006. Drawing graphs with dot.
* Goyal and Durrett (2019) Tanya Goyal and Greg Durrett. 2019. Embedding time expressions for deep temporal ordering models. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 4400-4406, Florence, Italy. Association for Computational Linguistics.
* 15, Pasadena, CA USA.
* Han et al. (2019) Rujun Han, I-Hung Hsu, Mu Yang, Aram Galstyan, Ralph Weischedel, and Nanyun Peng. 2019. Deep structured neural network for event temporal relation extraction. In _Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL)_, pages 666-106.
* He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In _Proceedings of the IEEE conference on computer vision and pattern recognition_, pages 770-778.
* Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jurgen Schmidhuber. 1997. Long short-term memory. _Neural computation_, 9(8):1735-1780.
* Holtzman et al. (2019) Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2019. The curious case of neural text degeneration. _arXiv preprint arXiv:1904.09751_.
* Karpathy (2015) Andrej Karpathy. 2015. The unreasonable effectiveness of recurrent neural networks. _Andrej Karpathy blog_, 21:23.

 

Chen Lin, Dmitriy Dligach, Timothy A Miller, Steven Bethard, and Guergana K Savova. 2016. Multilayered temporal modeling for the clinical domain. _Journal of the American Medical Informatics Association_, 23(2):387-395.
* Lin [2004] Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In _Text summarization branches out_, pages 74-81.
* Ling and Weld [2010] Xiao Ling and Daniel S Weld. 2010. Temporal information extraction. In _AAAI_, volume 10, pages 1385-1390.
* Liu et al. [2018] Zhengzhong Liu, Chenyan Xiong, Teruko Mitamura, and Eduard Hovy. 2018. Automatic event salience identification. _arXiv preprint arXiv:1809.00647_.
* Loshchilov and Hutter [2017] Ilya Loshchilov and Frank Hutter. 2017. Decoupled weight decay regularization. _arXiv preprint arXiv:1711.05101_.
* Luong et al. [2015] Minh-Thang Luong, Hieu Pham, and Christopher D Manning. 2015. Effective approaches to attention-based neural machine translation. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_, pages 1412-1421.
* Madaan et al. [2020] Aman Madaan, Dheeraj Rajagopal, Yiming Yang, Abhikasha Ravichander, Eduard Hovy, and Shrimai Prabhumoye. 2020. Eigen: Event influence generation using pre-trained language models. _arXiv preprint arXiv:2010.11764_.
* Makkonen et al. [2003] Juha Makkonen, Helena Ahonen-Myka, and Marko Salmenkivi. 2003. Topic detection and tracking with spatio-temporal evidence. In _European Conference on Information Retrieval_, pages 251-265. Springer.
* Newman [2004] Mark EJ Newman. 2004. Fast algorithm for detecting community structure in networks. _Physical review E_, 69(6):066133.
* Newman and Girvan [2004] Mark EJ Newman and Michelle Girvan. 2004. Finding and evaluating community structure in networks. _Physical review E_, 69(2):026113.
* Ning et al. [2019a] Qiang Ning, Zhili Feng, and Dan Roth. 2019a. A structured learning approach to temporal relation extraction. _arXiv preprint arXiv:1906.04943_.
* Ning et al. [2019b] Qiang Ning, Sanjay Subramanian, and Dan Roth. 2019b. An improved neural baseline for temporal relation extraction. _arXiv preprint arXiv:1909.00429_.
* Ning et al. [2020] Qiang Ning, Hao Wu, Rujun Han, Nanyun Peng, Matt Gardner, and Dan Roth. 2020. Torque: A reading comprehension dataset of temporal ordering questions. _arXiv preprint arXiv:2005.00242_.
* Ning et al. [2018] Qiang Ning, Ben Zhou, Zhili Feng, Haoruo Peng, and Dan Roth. 2018. Cogcomptime: A tool for understanding time in natural language. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_, pages 72-77.
* Papineni et al. [2002] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In _Proceedings of the 40th annual meeting on association for computational linguistics_, pages 311-318. Association for Computational Linguistics.
* Pennington et al. [2014] Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In _Empirical Methods in Natural Language Processing (EMNLP)_, pages 1532-1543.
* Radford et al. [2018] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pretraining. _URL https://s3-us-west-2. amazonaws. com/openai-assets/researchcovers/languageunsupervised/language understanding paper. pdf_.
* Radford et al. [2019] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. _OpenAI Blog_, 1(8):9.
* Raffel et al. [2019] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. _arXiv preprint arXiv:1910.10683_.
* Rajagopal et al. [2021] Dheeraj Rajagopal, Aman Madaan, Niket Tandon, Yiming Yang, Shrimai Prabhumoye, Abhilsha Ravichander, Peter Clark, and Eduard Hovy. 2021. Curie: An iterative querying approach for reasoning about situations.
* Recasens et al. [2010] Marta Recasens, Eduard H Hovy, and Maria Antonia Martti. 2010. A typology of near-identity relations for coreference (nident). In _LREC_.
* Sap et al. [2019] Maarten Sap, Ronan Le Bras, Emily Allaway, Chandra Bhagavatula, Nicholas Lourie, Hannah Rashkin, Brendan Roof, Noah A Smith, and Yejin Choi. 2019. Atomic: An atlas of machine commonsense for if-then reasoning. In _Proceedings of the AAAI Conference on Artificial Intelligence_, volume 33, pages 3027-3035.
* Sharma et al. [2017] Shikhar Sharma, Layla El Asri, Hannes Schulz, and Jeremie Zumer. 2017. Relevance of unsupervised metrics in task-oriented dialogue for evaluating natural language generation. _CoRR_, abs/1706.09799.
* Speer et al. [2017] Robyn Speer, Joshua Chin, and Catherine Havasi. 2017. Conceptnet 5.5: An open multilingual graph of general knowledge. In _Thirty-First AAAI Conference on Artificial Intelligence_.
* Sutskever et al. [2014] Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In _Advances in neural information processing systems_, pages 3104-3112.
* UzZaman and Allen [2012] Naushad UzZaman and James F Allen. 2012. Interpreting the temporal aspects of language. Citeseer.

 

* [V. V. L. Klimov, A. M. G. T. P. Nunes, and J. S. R. Hinton (2013) The role of the learning process in neural networks. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 496-508. Cited by: SS1.
* [A. V. L. Klimov, A. M. G. T. P. Nunes, and J. S. R. Hinton (2013) Learning to be efficient. In Advances in Neural Information Processing Systems, pp. 1-10. Cited by: SS1.
* [S. Vashishtha, B. Van Durme, and A. Steven White (2019) Fine-grained temporal relation extraction. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, pp. 2906-2919. External Links: Link, Document Cited by: SS1.
* [Y. You, R. Ying, X. Ren, W. L. Hamilton, and J. Leskovec (2018) Graphrnn: generating realistic graphs with deep auto-regressive models. arXiv preprint arXiv:1802.08773. Cited by: SS1.

 

Learning Event Communities Using Community Detection

In this section, we provide the details on the community detection algorithm used by our method. We define the temporal event communities to be a division of the temporal graph \({\bf G}({\bf V},{\bf E})\) into sub-graphs \({\bf G_{1}}({\bf V_{1}},{\bf E_{1}}),{\bf G_{2}}({\bf V_{2}},{\bf E_{2}}),...,{ \bf G_{K}}({\bf V_{k}},{\bf E_{k}})\) such that the events in a community (sub-graph) \({\bf G_{i}}\) are more co-referential to each other as opposed to the other events in the temporal graph. We use the undirected link between two events \(e_{j},e_{i}\) as a proxy for them being co-referential, and learn temporal event communities utilizing the concept of modularity, first introduced by Newman and Girvan (2004).

Formally, let \({\bf A}\) be the undirected adjacency matrix for a temporal graph \({\bf G}({\bf V},{\bf E})\) such that \({\bf A}(e_{i},e_{j})=1\) if \(e_{i}\) and \(e_{j}\) are connected by a temporal relation, and 0 otherwise. Further, let \(\delta(e_{i},e_{j})=1\) if events \(e_{i},e_{j}\) belong to the same temporal community, and \(0\) otherwise. For a given \(\delta\), we denote the fraction of the edges that exist between events that belong to the same communities by \(f_{same}=\frac{\sum_{e_{i},e_{j}\in{\bf E}}{\bf A}(e_{i},e_{j})\delta(e_{i}, e_{j})}{2|{\bf E}|}\). Where the \(2|{\bf E}|\) in the denominator is due to the fact that \({\bf A}\) treats \({\bf G}\) as an undirected graph. Let the popularity \(p\) of an event \(e_{i}\) be the number of events that are linked to it i.e. \(p(e_{i})=\sum_{e_{j}\in{\bf E}}{\bf A}(e_{i},e_{j})\). The probability of randomly picking an event \(e_{i}\) when sampled by popularity is \(\frac{p(e_{i})}{\sum_{e_{j}\in{\bf E}}p(e_{i})}=\frac{p(e_{i})}{2|{\bf E}|}\). Thus, if edges are created randomly by sampling nodes by popularity \(p\) of the nodes, the fraction of edges within the communities, \(f_{random}\), is given by

\[f_{random}=\frac{\sum_{e_{i},e_{j}\in{\bf E}}p(e_{i})p(e_{j})\delta(e_{i},e_{ j})}{2|{\bf E}|*2|{\bf E}|}\]

Finally, defining modularity, \(Q\), to be \(f_{same}-f_{random}\):

\[Q=\frac{1}{2|{\bf E}|}*\sum_{e_{i},e_{j}\in{\bf E}}{\bf A}(e_{i},e_{j})-\frac {p(e_{i})p(e_{j})\delta(e_{i},e_{j})}{2|{\bf E}|}\]

We want to learn community assignments \(\delta\) that maximize \(Q\). A high \(Q\) would promote \(f_{same}>f_{random}\) and thereby encourage highly inter-connected event communities. Calculating such \(\delta\) directly is not tractable, since the complexity of such an operation would be at least exponential in the number of events Newman (2004). We use the fast implementation provided by Clauset et al. (2004) for calculating event communities iteratively. The algorithm converges at \(Q\)\(0.3\). We use a similar approximation at test time: given a document \(D\), we first break it down into sub-documents using caevo and then feed each sub-document to our method.

## Appendix B Using a smaller block size

We found that the performance drops when using a block size of 300 and batch size of 2. Table 10 presents the results.

### Masked Language Modeling Using Transformers

In this section, we expand on the design of the transformer blocks. For ease of reference, we reiterate our training methodology. We train a (separate) conditional language model to solve both the tasks. Specifically, given a training corpus of the form \(\{(\mathbf{x_{i}},\mathbf{y_{i}})\}\), we aim to estimate the distribution \(p_{\theta}(\mathbf{y_{i}}\mid\mathbf{x_{i}})\). Given a training example \((\mathbf{x_{i}},\mathbf{y_{i}})\) we set \(\mathbf{u_{i}}=\mathbf{x_{i}}\lVert\mathbf{y_{i}}\).8\(p_{\theta}(\mathbf{u_{i}})\) can then be factorized as a sequence of auto-regressive conditional probabilities using the chain rule: \(p_{\theta}(\mathbf{u_{i}})=\prod_{k=1}^{n}p(u_{i,k}|\mathbf{u_{i}}_{<\mathbf{k}})\), where \(u_{i,k}\) denotes the \(k^{th}\) token of the \(i^{th}\) sequence, and \(\mathbf{u_{i}}_{<\mathbf{k}}\) denotes the sequence of tokens \(\{u_{1},u_{2},...,u_{k-1}\}\). Language models are typically trained by minimizing a cross-entropy loss \(-\text{log}p_{\theta}(\mathbf{u_{i}})\) over each sequence \(\mathbf{u_{i}}\) in \({\bf X}\). However, the cross-entropy loss captures the joint distribution \(p_{\theta}(\mathbf{x_{i}},\mathbf{y_{i}})\), and is not aligned with our goal of learning conditional distribution \(p_{\theta}(\mathbf{y_{i}}|\mathbf{x_{i}})\). To circumvent this, we train our model by masking the loss terms corresponding to the input \(\mathbf{x_{i}}\), similar to Bosselut et al. (2019). Let \(\mathbf{m_{i}}\) be a mask vector for each sequence \(\mathbf{u_{i}}\), set to \(0\) for positions corresponding to \(\mathbf{x_{i}}\), and \(1\) otherwise

\begin{table}
\begin{tabular}{c c c c c} \hline \hline bleu & mtr & rg & dot\% \\ \hline
**25.01** & **27.95** & **60.99** & **91.71** \\ \hline \hline \(v_{P}\) & \(v_{R}\) & \(v_{F_{1}}\) & \(e_{P}\) & \(e_{R}\) & \(e_{F_{1}}\) \\ \hline
70.31 & 64.75 & 65.68 & 29.43 & 24.83 & 24.27 \\ \hline \end{tabular}
\end{table}
Table 10: Results for TG-Gen using a block size of 300 and a block size of 2.

 

i.e. \(m_{i,j}=1\) if \(j>|\mathbf{x}_{i}|\), else 0. We combine the mask vector with our factorization of \(p_{\theta}(\mathbf{u}_{i})\) to formulate a _masked_ language modeling loss, which is minimized over the training corpus \(\mathbf{X}\) to estimate the optimal \(\theta\):

\[\mathcal{L}_{masked}(\mathbf{X})=-\sum_{i=1}^{|\mathbf{X}|}\sum_{ j=1}^{|x_{i}|+|y_{i}|}m_{i,j}*\text{log}(p_{\theta}(u_{i,j}|\mathbf{u_{i,<j}}))\]

Note that the formulation of masked loss is opaque to the underlying architecture, and can be implemented with a simple change to the loss function. Intuitively, the model is optimized for only the output sequence \(y_{i}\).

### Adapting gpt-2 for Masked Language Modeling

In practice, we use gpt-2Radford et al. (2019) based on transformer architecture Vaswani et al. (2017) for our implementation. An input sequence \(\mathbf{u_{i}}\) of length \(n\) is first embedded to a continuous representation denoted by \(\mathbf{u_{i}}^{(0)}\in\mathbb{R}^{nd}\). \(\mathbf{u_{i}}^{(0)}\) is then passed through a series of \(L\)_transformer blocks_ to obtain the output sequence \(\mathbf{u_{i}}^{(L)}\in\mathbb{R}^{nh}\). Each transformer block Vaswani et al. (2017) consists of two operations: an auto-regressive version of the multiheaded self-attention Vaswani et al. (2017) operation (_AutoRegMultiHead_) followed by a feed-forward operation (_FFN_). Each of these operations is surrounded by a residual connection He et al. (2016) and followed by a layer normalization Ba et al. (2016) operation. Denoting by \(\mathbf{u}^{(l-1)}\) the input to the \(l^{th}\) transformer block , the operations are in a transformer block are defined as follows:

\[\tilde{\mathbf{u}}_{attn}^{l} =\textit{AutoRegMultiHead}(\mathbf{u}^{(l-1)})\] \[\mathbf{u}_{att}^{(l)} =\textit{LayerNorm}(\tilde{\mathbf{u}}_{att}^{(l)}+\mathbf{u}^{(l-1)})\] \[\tilde{\mathbf{u}}_{ffn}^{(l)} =\textit{FFN}(\mathbf{u}_{att}^{(l)})\] \[\mathbf{u}^{(l)} =\textit{LayerNorm}(\tilde{\mathbf{u}}_{ffn}^{(l)}+\mathbf{u}_{att}^{(l)})\]

Where _AutoRegMultiHead_ is an auto-regressive version of the multiheaded self-attention Vaswani et al. (2017) that restricts the attention to the sequence seen so far (in accordance with the chain rule), and _FFN_ is a feed-forward network (mlp). After obtaining \(\mathbf{u_{i}}^{(L)}\), we set \(p_{\phi}(\mathbf{u_{i}})=\text{softmax}(\mathbf{u_{i}}^{(L)}*\textbf{W}_{e})\), where \(\textbf{W}_{e}\in\mathbb{R}^{h|V|}\) (\(|V|\) is the size of the vocabulary). Finally, we calculate the masked loss as \(\mathcal{L}(\mathbf{u_{i}})=\mathbf{m_{i}}^{T}\odot\text{log}(p_{\phi}(\mathbf{u_{i}}))\), and the optimal \(\phi\) is obtained by minimizing \(\mathcal{L}_{masked}(\mathbf{X})=-\sum_{i=1}^{|\mathbf{X}|}\mathcal{L}(\mathbf{u_{ i}})\).

## Appendix D Dataset Statistics

Tables 11, 12, and 13 list various statistics calculated from the source data.

 

They were parents and grandparents who thought they had fully grasged the perils facing their teenagers in the touch working-class streets north of Kennedy Airport in Queens . They said that they understood the power of peers , the lurse of gang and drops, the impact of movies and music and television that with relentless repetition depicted a culture of cassal violence , And ; they said , they believed they bad taken the necessary precautions . They gave their children beepers and cellular phones to keep better track of them . They lined up mentors , and set cutfrows , and encouraged faith .

\begin{table}
\begin{tabular}{l c c} \hline \hline Relation & Raw Frequency & \% Frequency \\ \hline before & 2436201 & 54.51 \\ after & 1772071 & 39.65 \\ is included & 131052 & 2.93 \\ simultaneous & 112509 & 2.52 \\ includes & 17465 & 0.39 \\ \hline \hline \end{tabular}
\end{table}
Table 13: Relation Frequence in our Corpus

Figure 4:



In the last month , Eastman Kodak , which has been shrinking for the last decade , announced plans to lay off an additional 6,000 to 9,000 workers . The Rochester police would to step up efforts after it reported a homicide rate that was the worst in years . Candidates competing for the region 's top offices are holding regular news conferences to criticise one another .

Figure 5:



the dream of being a biochemical and revexi in science fiction novels , insect collecting and dismantling broken electronic equipment . He acknowledges that his actions were potentially dangerous : people can't see the difference between showing off , testing the boundaries and killingly people ; he said . I went into the school to see what I could get away with , not because i wanted to kill anyone . Michael said the most difficult part of his punishment was the six weeks he spent at the Essex County Youth House in Dewat . One cellate , be said , best him and tried to strangle him .

Figure 8:

Figure 7:



An article yesterday about the release of an Egyptian chemist who had been held by Egyptian police as a suspect in the July 7 London transit bombings misstated the date he was arrested in Cairo . It was July 14 , not June 14 .

Figure 9