SlowGuess/NAACL_2022_OCR huggingface dataset:
Puneet Mathur‡, Vlad I Morariu†, Verena Kaynig-Fittkau†, Jiumiang Gu†, Franck Dernoncourt†, Quan Hung Tran†, Ani Nenkova†,Dinesh Manocha‡, Rajiv Jain† ‡ University of Maryland, College Park † Adobe Research ‡{puneetm, dmanocha} @umd.edu ‡{morariu, kaynigfi, jigu, dernonco} @adobe.com ‡{qtran, nenkova, rajijain} @adobe.com
We introduce DocTime - a novel temporal dependency graph (TDG) parser that takes as input a text document and produces a temporal dependency graph. It outperforms previous BERT based solutions by a relative  $4 - 8\%$  on three datasets from modeling the problem as a graph-network with path-prediction loss to incorporate longer range dependencies. This work also demonstrates how the TDG graph can be used to improve the downstream tasks of temporal questions answering and NLI by a relative  $4 - 10\%$  with a new framework that incorporates the temporal dependency graph into the self-attention layer of Transformer models (Time-transformer). Finally, we develop and evaluate on a new temporal dependency graph dataset for the domain of contractual documents, which has not been previously explored in this setting.
Understanding the temporal relations between events mentioned in a document is an important natural language task with applications in downstream tasks such as timeline creation (Leeuwenberg and Moens, 2018), time-aware summarization (Noh et al., 2020), temporal question-answering (Ning et al., 2020a), and temporal information extraction (Leeuwenberg and Moens, 2019). This area of research remains important yet challenging due to several limitations such as confounded modalities (eg. events that are certain to happen vs the ones that might happen), event ambiguity (eg. agreeing to terms of a contract vs signing a contract) and need for complete annotation of all event pairs for precise temporal localization (Yao et al., 2020a).
Early work densely annotated all pairs of events to address this problem (Cassidy et al., 2014), but was limited to short passages or adjacent sentences due to the  $\binom{n}{2}$  complexity of the task, especially for long documents. Recently this problem formulation was significantly simplified using temporal
dependency trees (TDT) (Zhang and Xue, 2019) and temporal dependency graphs (TDG) (Yao et al., 2020a) by only capturing the reference TIMEX or event to build a dependency graph to capture this information. This enabled the development of temporal dependency parsers (Zhang and Xue, 2018a; Ross et al., 2020a) to infer temporal relationships more robustly and efficiently.
We introduce DocTime - a state-of-the-art temporal dependency parser that parses document-level text to produce temporal dependency graphs. Unlike previous approaches using contextual features such as BERT(Ross et al., 2020b), our model utilizes a graph network and a novel path prediction loss to reason over long-range multi-hop dependencies while maintaining global consistency of temporal ordering of inter-dependent events.
To validate the utility of DocTime and our generated temporal dependency graph, we go one step further than prior work and explore the question of whether temporal dependency graphs are useful for downstream tasks by introducing Time-Transformer. It is a framework to incorporate temporal dependency graphs into existing transformer-based architectures without retraining from scratch. We demonstrate the usefulness of our proposed Time-Transformer on temporal NLI (Vashishtha et al., 2020) and time-sensitive question answering (Chen et al., 2021) tasks.
Prior work on temporal relationship extraction and temporal dependency parsing have been mostly limited to news (Zhang and Xue, 2019; Yao et al., 2020a; Pustejovsky et al., 2003a), narrative stories (Zhang and Xue, 2018b; Kolomiyets et al., 2012) or clinical notes (Bethard et al., 2016). In addition to experimenting with existing temporal dependency parsing datasets, we introduce a dataset for temporal dependency graphs in a new domain - contractual documents, where temporal reasoning over events has real world legal and monetary implications for users.
Human Language Technologies, pages 993 - 1009
Figure 1: DocTime encodes rich token level embeddings from input document using structural, syntactic, and semantic graphs through BERT-GCN, WR-GCN and HyperGraph Conv layers, respectively. Token-level features are concatenated and passed through Iterative Deep Graph Learning (IDGL) to learn a noisy dependency structure over the TIMEX and Event entities. Graph U-net allows the model to incorporate longer range dependencies for predicting the final temporal dependency graph structure and relationships. The model is trained with a novel auxiliary path prediction loss to learn multi-hop connections in TDG.
Our main contributions include:
- A novel document-level temporal dependency parser (DocTime) that predicts the temporal dependency graph from text in an end-to-end manner with a novel path prediction loss, which outperforms the current SOTA by a relative  $4 - 8\%$  on three datasets.
- Time-Transformer, a novel framework to incorporate Temporal Dependency Graphs into transformer models for downstream tasks without needing to retrain from scratch. Results on natural language inference and question answering with a new self-attention module show a relative  $4\% - 10\%$  improvement.
- Development of new document-level ( $>1500$  words) TDG dataset in the domain of contractual documents (ContractTDG $^1$ ).
Temporal Dependency Parsing: Previous work has been devoted to pairwise classification of relations between events and time expressions, notably TimeBank (Pustejovsky et al., 2003b) and its extensions like Cassidy et al. (2014) annotated all relations. Pair-wise annotation have multiple problems including polynomial square complexity, global inconsistencies in predictions due to relation transitivity and forced annotation of vague relations (Ning et al., 2018). Prior work focuses on extracting temporal relations between event pairs in the same sentence or adjacent sentences (Goyal and Durrett, 2019; Ning et al., 2019a; Han et al., 2019a,c,b, 2020; Ballesteros et al., 2020; Zhao et al., 2020). TIMERS Mathur et al. (2021a) presented temporal relation extraction in long document.
Temporal Dependency Parsing (TDP): Tem-
poral dependency trees were first proposed by Kolomiyets et al. (2012). (Zhang and Xue, 2018b) provided the earliest TDT corpus on news data and narrative stories, (Zhang and Xue, 2019) released the first English TDT corpus. Yao et al. (2020a) relaxed the assumption of single reference edge in dependency trees to form the improved TDG. (Zhang and Xue, 2018a) built an end-to-end neural temporal dependency parser using BiLSTM and Ross et al. (2020b) improved it further incorporating BERT. Our approach improves by modeling complex dependencies and introduces a new resource for TDG in contracts.
Linguistically-aware Transformers: Recent works have investigated using linguistic features as a prior for Transformer models. Syntax-bert (Bai et al., 2021a) uses syntactic and constituency dependency on NLI and GLUE benchmarks. Coreference-Informed Transformer (Liu et al., 2021) performs coreference-aware dialogue summarization. Temporal reasoning about event ordering can find applications in many tasks such as summarization (Noh et al., 2020), question answering (Chen et al., 2021; Ning et al., 2020b; Jin et al., 2020), commonsense reasoning (Qin et al., 2021), and natural language inference (Vashishtha et al., 2020). We propose to use TDG as priors to Transformer models to make them temporally-aware for use in downstream tasks.
3 DocTime: Document TDG Parsing
Task Formulation: Let document  $D$  be defined as a sequence of  $n$  tokens  $[x_1, \dots, x_n]$ . The entire document can be seen as a sequence of  $m$  sentences  $[s_1, \dots, s_m]$ . Each document has a set of  $p$  events  $E = [e_1, \dots, e_p]$  and  $q$  times  $T = [t_1, \dots, t_q]$ , where  $p, q \leq n$ . The creation date of the document is represented by timestamp  $t_{DCT}$ . Yao
1https://github.com/contractTDG/ ContractTDG_Dataset
et al. (2020a) defines a temporal dependency graph (TDG) where each timex node always has a reference timex, which is the most specific narrative time related to the event (Pustejovsky and Stubbs, 2011). If such a narrative time is not available, the timex should be anchored to the DCT. An event node can either have a reference timex or be connected to a reference event, which is an event that provides the most specific temporal location. The task of temporal dependency graph parsing of a text document  $D$  results in a dependency graph  $G = (C,V)$ , where  $C$  represents the set of all events, timexes and the document creation date (DCT).  $V$  is the set of all edges in the graph, where each edge represents a temporal relationship  $\Re$  between corresponding entity node pair  $V = \{(t_i,t_j),(e_i,e_j),(e_i,t_j)\} \forall i,j\in C$ .
Model Overview: Figure 1 shows an overview of our network architecture for temporal dependency parsing. We first extract token level BERT features from the input document, which are then enriched by three graph networks that encode structural, syntactic, and semantic relationships. This is followed by Iterative Deep Graph Learning over the TIMEX and Event entities to learn an initial dependency structure. This is passed through a Graph U-net to allow the model to incorporate longer range dependencies before predicting the final temporal dependency graph and relationships. The model is also trained with a novel auxiliary path prediction loss.
3.1 Feature Encoding
We leverage the pre-trained BERT language model to obtain the embeddings for each token as follows:  $w_{1},w_{2},\dots ,w_{n} = \mathrm{BERT}([x_{1},x_{2},\dots ,x_{n}])$  where  $w_{i}$  is the embedding of the token  $x_{i}$ . As document sequence lengths can be larger than 512, we use a sliding window encoding technique to encode whole documents. We average the embeddings of overlapping tokens of different windows to obtain the final representations. These token representations are enriched with slightly enhanced variants of the structural  $(G_{str})$ , syntactic  $(G_{syn})$  and semantic  $(G_{sem})$  graphs utilized by (Mathur et al., 2021b) for document-level temporal relationship extraction. The key differences are the use of BERT-GCN (Lin et al., 2021) to combine contextual and structural graph features, the addition of co-reference relationships to the syntactic graph, and the use of a hypergraph convolution (Bai et al., 2021b) to allow for token level features in the se
mantic graph. All aspects of these features and the changes are presented in Appendix B.
3.2 Temporal Dependency Prediction
We combine the learned representation for each entity node (timex, event, DCT) by concatenating the node embeddings learned from structural, syntactical and semantic graphs to obtain a D-dimensional feature vector for each of  $z$  entities in the document given by  $\mathbf{F}(w_i) = g_i^{str} \oplus g_i^{syn} \oplus g_i^{sem}$ , where  $\oplus$  represents concatenation. We retain only the enriched node embeddings for each word. We then utilize Iterative Deep Graph Learning (IDGL) $^2$  (Chen et al., 2020) to dynamically learn an initial dependency graph structure from the combined node embeddings. Given a noisy graph input feature matrix  $\mathbf{F} \in R^{l*D}$ , IDGL produces an implicitly learned graph structure  $G^* = \{A^*, \mathbf{F}, F_l\}$  with a jointly refined corresponding graph node embeddings  $\mathbf{F}'$  with adjacency matrix  $A^*$  by optimizing with respect to downstream link prediction task  $F_l$  between entity nodes.
3.2.1 Graph U-net For Higher Level Features
The Graph U-net (Gao and Ji, 2019) is a U-shaped graph encoder-decoder architecture containing two down-sampling graph pooling (gPool) layers and two up-sampling graph unpooling (gUnpool) layers with skip connections. gPool layers reduce the size of the graph to encode higher-order features, while the gUnpool layer restores the graph into its higher resolution structure, thereby promoting information exchange between entity pairs through an enlarged receptive field. Each graph pooling and unpooling layer is followed by a GCN layer to implicitly capture the topological information in the input graph. Taking the dynamically learned graph structure  $G^{*}$ , a graph embedding layer converts input node features  $\mathbf{F}$  into low-dimensional representations that are then passed through a graph U-net encoder-decoder  $\mathcal{U}$  to acquire entity-level relation matrix  $\mathbf{Y} = \mathcal{U}(\mathbf{F})$ ,  $\mathbf{Y} \in R^{l * l * D'}$ .
3.2.2 Temporal Dependency Link Prediction and Relation Classification
Given entity adjacency matrix  $A^{*}$  and entity-level relation matrix  $\mathbf{Y}$ , we use a bilinear function to map them to link and relation probabilities  $\mathbf{Z}_l$  and  $\mathbf{Z}_r$ , respectively. Formally, we have  $\mathbf{Z}_l = \sigma (\mathbf{Y}W_l\mathbf{Y} + b_l)$  and  $\mathbf{Z}_r = \sigma (A^* W_rA^* +b_r)$ , where
2Implementation: graph4ai/graph4nlp
Figure 2: Time-Transformer is a variant of pre-trained Transformer models that augments temporal knowledge into the self-attention layer during fine-tuning of the Transformer model on different downstream tasks. Input text is converted into a temporal dependency graph using DocTime parser. The graph is then converted into a set of masks that encodes the temporal relationship between each token (i.e. After, Before) using the novel Temporally- informed Self-Attention (TISA). TISA creates K masks to represent the (k)-hop distance between two nodes in TDG for aggregating information across longer ranges in the input. TISA uses hyperbolic feed-forward layer to learn the mask weights.
$W_{l}, W_{r}, b_{l}, b_{r} \in R^{D' * D'}$  represent learnable parameters. This is followed by a Softmax layer for link prediction and relations classification.
3.3 Training DocTime
**Path Reconstruction Loss:** In a document-level temporal parsing setup, the majority of node pairs may not have any ground truth link or temporal relation. Graph representation learning methods universally model relations between all entity pairs regardless of whether the entity pair has any relationship, leading to dispersion of attention in learning most non-existent edge connections. We propose path reconstruction loss  $L_{path}$ , which forces the model to pay more attention to learn entity pairs with relationships rather than ones without relationships. Equation 1 gives the cross entropy loss over all direct edge connection between all pairs of entities, where  $r_j^i$  indicates the relation between the entity pair and  $P(r_j^i)$  is probability of relation label  $r$ . Path reconstruction loss  $L_{path}$  modifies the cross entropy loss  $L_{ce}$  function as shown in Equation 2 by sampling all  $n^2$  entity pairs and maximizing the probability of the shortest dependency path  $\mathcal{N}(\phi)$  between the entity pair nodes. Finally, the path reconstruction loss and the existing classification loss are added as the training objective for DocTime, given by  $L = L_{path} + L_{ce}$ .
$$
\begin{array}{l} L _ {c e} = - \frac {1}{\sum_ {i = 0} ^ {l} N _ {i}} \sum_ {i = 1} ^ {l} \sum_ {j = 1} ^ {N _ {i}} \{r _ {j} ^ {i} \log P (r _ {j} ^ {i}) \\ \left. + \left(1 - r _ {j} ^ {i}\right) \log \left(1 - P \left(r _ {j} ^ {i}\right)\right) \right\} \tag {1} \\ \end{array}
$$
$$
\begin{array}{l} L _ {p a t h} = - \frac {1}{\sum_ {i = 0} ^ {l} N _ {i}} \sum_ {i = 1} ^ {l} \sum_ {j = 1} ^ {N _ {i}} \{r _ {j} ^ {i} \log \mathcal {N} (\phi_ {i}) \\ \left. + \left(1 - r _ {j} ^ {i}\right) \log \left(1 - \mathcal {N} \left(\phi_ {i}\right)\right) \right\} \tag {2} \\ \end{array}
$$
Multi-task Training: Dependency link prediction and entity-level relation classification are correlated tasks and reinforce each other. We use multi-task training to optimize both tasks simultaneously using the path prediction cross entropy loss. The final optimization uses a weighted sum of the dependency link prediction loss and entity-level relation classification loss  $L = \lambda L_{l} + (1 - \lambda)L_{r}$ , where the weighting factor  $\lambda$  is a hyperparameter.
4 Time-Transformer
We would also like to understand our temporal dependency parsing can be useful for downstream tasks requiring temporal reasoning. Here we introduce the Time-Transformer, which allow a TDG generated by DocTime to be combined with state-of-the-art transformer models for temporal tasks. The Time-Transformer augments the flow of information in a Transformer network via a temporally-informed self-attention mechanism. We first for
mulate the Time-Transformer architecture in  $\S 4$  and then construct of temporally-informed attention layers in  $\S 4$ .
Architecture: Time-Transformer was motivated by recent work incorporating syntax (Bai et al., 2021a) or co-reference graphs(Liu et al., 2021) into the transformer architecture to improve downstream tasks. In each case, these approaches encode additional knowledge from the sparse graphs as a masked self attention layer into the transformer. Figure 2 shows the architecture of Time-Transformer incorporating temporal knowledge into the self-attention layer during finetuning of the Transformer model. Input text is converted into a temporal dependency graph using DocTime parser. The graph is then converted into a set of masks that encodes the temporal relationship between each entity (i.e. After) explained in more detail in the next section: Temporally-informed Self-Attention. The input embedding (token+positional+attention masks) is passed through the Time-Transformer model which modifies the self-attention layer of the standard Transformer architecture with a temporally-informed self-attention layer to be fine-tuned on downstream tasks.
TISA: Temporally-informed Self-Attention : The TDG produced by DocTime is sparse and to effectively utilize the graph extracted by the temporal dependency parser for longer range temporal relationships, we utilize  $K$  self-attention layers that encode the temporal relationship if traversing  $K$  hops in the TDG as shown in 2. More formally starting from node  $A$ , the minimum number of hops  $(k)$  required to reach another node  $B$  can be regarded as k-hop distance between  $A$  and  $B$ , written as  $k$ -hop  $(A, B)$ . We create  $K$  masks to represent the  $(k)$ -hop distance between two nodes to allow the model to aggregate information across longer ranges in the TDG. Specifically, a mask  $M \in \{0, 1, 2, \dots, r\}^{n \times n}$  denotes if there is a relation between entity  $i$  and  $j$ , and  $n$  is the number of tokens in the input text. The value of the mask is the relationship type for  $i$  and  $j$ . It is found by inferring the relationship using Allen's interval algebra (Allen, 1983) and is set to 0 if there is no relationship or set to "Overlap" if there is a conflict. We adopt a soft-mask learning strategy to enable the self-attention layer to re-weight the importance of each mask and avoid the problem of vanishing gradient. A hyperbolic feed-forward layer is used
to learn the mask weights as research has shown it can avoid distortion of the feature space in graph representations (Ganea et al., 2018). The value of  $K$  is a hyperparameter that can be customized according to the nature of input dependency graph.
Training Time-Transformer: For each dataset, we optimize the hyper-parameters of Time-Transformer through grid search on the validation data. In all our experiments, we limit the maximum value of  $k$ -hop to 15. Detailed settings can be found in the appendix.
5.1 Temporal Graph Parsing Datasets
We train and evaluate DocTime on three datasets. First is the Temporal Dependency Graphs (TDG) dataset (Yao et al., 2020a) made up of 500 Wikinews articles annotated with document-level temporal dependency graphs. Second is the Temporal Dependency Trees (TDT) dataset Zhang and Xue (2019) made from 183 documents derived from TimeBank (Pustejovsky et al., 2003a) annotated with a temporal dependency tree structure. The third dataset we created as part of this paper and is describe in more detail below.
Contract-TDG: Understanding the temporal relationship of events in contracts is an important business problem, where understanding event timelines can have legal and monetary consequences. Previous work on temporal relationships has largely focused on clinical, news or narrative text, whereas to the best of our knowledge the contractual domain has not been explored for this problem. To construct this dataset, we used 100 contracts from the Atticus contracts dataset $^{3}$  (Hendrycks et al., 2021), which were sourced from public domain SEC contracts. Due to the multi-page length of these documents, we limited the annotations to the first 1500 words. We did not include definition sections, since they did not contain many events of interest for this task. The documents have a 70-10-20 split for training, validation, and testing.
To obtain the TDG annotations required for our task, we followed the 5 steps procedure outlined by the original TDG dataset in (Yao et al., 2020b): (i) TIMEX Identification (TE), (ii) Identifying reference times for TE, (iii) Event identification, (iv) Identifying reference times for events, (v) Identifying reference events for events. Document Creation
<sup>3</sup>https://www.atticusprojectai.org/cuad
Figure 3: Example of a temporal dependency graph from ContractTDG dataset annotated using Brat Tool.
Dataset	Docs	Timex	Events	Refs
TimeBank (Pustejovsky et al., 2003b)	183	1,414	7,935	6,148
TB-Dense (Cassidy et al., 2014)	36	289	1,729	12,715
MATRES (Ning et al., 2019b)	275	-	1,790	13,577
TDT-Crd (Zhang and Xue, 2019)	183	1,414	2,691	4,105
TDG (Yao et al., 2020a)	500	2,485	14,974	28,350
Contract-TDG (Ours)	100	2354	11,752	12,909
Table 1: Comparison of ContractTDG data statistics to other temporal relation datasets. ContractTDG has fewer documents but comparable number of TIMEX/Events/relations.
Task	TDG(F1)	Contract TDG(F1)
1: TIMEX ID	0.96	0.93
2: TIMEX RT	0.89	0.81
3: Event ID	0.79	0.76
4: RT ID (U)	0.67	0.83
4: RT ID (L)	0.61	0.75
5: RE ID (U)	0.59	0.85
5: RE ID (L)	0.52	0.79
Table 2: Inter-Annotator Agreement (IAA) for the Contract-TDG and TDG dataset. U = structure, L = structure + labels
Times (DCT) were provided as effective dates in the ATTICUS corpus.
Similar to (Yao et al., 2020b) for tasks 1 (TE) and 3 (Event ID), we used the Mechanical Turk platform to obtain two annotations to validate text spans of noisy TIMEXes extracted by HeidelTime software $^{4}$  (Strötgen and Gertz, 2013) and verbs that were possible events. Disagreements were resolved by an expert annotator. However, for the reference tasks, we decided against using Mechanical Turk due to the difficulty and length of the contracts as well as the lower agreement faced by the original TDG system for the last two tasks. We instead used the BRAT annotation tool $^{5}$  (Stenetorp et al., 2012) with an expert annotator for tasks 2,4, and 5, following the (Yao et al., 2020b) guidelines. ContractTDG is annotated for four temporal relations - after, before, overlaps, and includes.
Table 1 compares the data statistics of the ContractTDG to previous temporal relationship and
temporal dependency corpora. Even though this dataset has many fewer documents than the TDG dataset, it has a large number of TIMEX, Events, and Temporal relationships due to the document length. Table 2 reports the F1 IAA metrics for ContractTDG dataset to directly compare to the original TDG dataset. For Tasks 1 and 3 we report IAA F1 for the two crowd sourced worker annotations and for the relationship tagging tasks (2,4,5), we report IAA metrics calculated on the test position (20% of the data) that was reviewed by two experts. The agreement is slightly lower for the TIMEX/Event identification tasks but higher for the three relationship tasks. We evaluate DocTime for dependency structure as well as structure+relation prediction for both development and test splits.
5.2 Time-Transformer Experiments for Downstream Tasks