SlowGuess/NAACL_2022_OCR huggingface dataset:
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019a. Roberta: A robustly optimized bert pretraining approach. ArXiv, abs/1907.11692.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019b. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Zhengyuan Liu, Ke Shi, and Nancy F Chen. 2021. Coreference-aware dialogue summarization. arXiv preprint arXiv:2106.08556.
Puneet Mathur, Meghna Ayyar, Sahil Chopra, Simra Shahid, Laiba Mehnaz, and Rajiv Shah. 2018a. Identification of emergency blood donation request on twitter. In Proceedings of the 2018 EMNLP Workshop SMM4H: The 3rd Social Media Mining for Health Applications Workshop & Shared Task, pages 27-31.
Puneet Mathur, Rajiv Jain, Franck Dernoncourt, Vlad Morariu, Quan Hung Tran, and Dinesh Manocha. 2021a. TIMERS: Document-level temporal relation extraction. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 524-533, Online. Association for Computational Linguistics.
Puneet Mathur, Rajiv Jain, Franck Dernoncourt, Vlad Morariu, Quan Hung Tran, and Dinesh Manocha. 2021b. Timers: Document-level temporal relation extraction. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 524-533.
Puneet Mathur, Ramit Sawhney, Shivang Chopra, Maitree Leekha, and Rajiv Ratn Shah. 2020. Utilizing temporal psycholinguistic cues for suicidal intent estimation. In European Conference on Information Retrieval, pages 265-271. Springer.
Puneet Mathur, Rajiv Shah, Ramit Sawhney, and Debanjan Mahata. 2018b. Detecting offensive tweets in Hindi-english code-switched language. In Proceedings of the Sixth International Workshop on Natural Language Processing for Social Media, pages 18-26.
Rohan Mishra, Pradyumn Prakhar Sinha, Ramit Sawhney, Debanjan Mahata, Puneet Mathur, and Rajiv Ratn Shah. 2019. Snap-batnet: Cascading author profiling and social network graphs for suicide ideation detection on social media. In Proceedings of the 2019 conference of the North American Chapter of the association for computational linguistics: student research workshop, pages 147-156.
Trisha Mittal, Puneet Mathur, Aniket Bera, and Dinesh Manocha. 2021. Affect2mm: Affective analysis of multimedia content using emotion causality. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5661-5671.
Qiang Ning, Sanjay Subramanian, and D. Roth. 2019a. An improved neural baseline for temporal relation extraction. In EMNLP/IJCNLP.
Qiang Ning, Sanjay Subramanian, and Dan Roth. 2019b. An improved neural baseline for temporal relation extraction. arXiv preprint arXiv:1909.00429.
Qiang Ning, Hao Wu, Rujun Han, Nanyun Peng, Matt Gardner, and Dan Roth. 2020a. Torque: A reading comprehension dataset of temporal ordering questions. In EMNLP.
Qiang Ning, Hao Wu, Rujun Han, Nanyun Peng, Matt Gardner, and Dan Roth. 2020b. Torque: A reading comprehension dataset of temporal ordering questions. arXiv preprint arXiv:2005.00242.
Qiang Ning, Hao Wu, and Dan Roth. 2018. A multi-axis annotation scheme for event temporal relations. In ACL.
Yunseok Noh, Yong-Min Shin, Junmo Park, A.-Yeong Kim, Su Jeong Choi, Hyun-Je Song, Seongbae Park, and Seyoung Park. 2020. Wire: An automated report generation system using topical and temporal summarization. Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval.
James Pustejovsky, Patrick Hanks, Roser Sauri, Andrew See, Robert Gaizauskas, Andrea Setzer, Dragomir Radev, Beth Sundheim, David Day, Lisa Ferro, et al. 2003a. The timebank corpus. In Corpus linguistics, volume 2003, page 40. Lancaster, UK.
James Pustejovsky, Patrick Hanks, Roser Saurí, Andrew See, Rob Gaizauskas, Andrea Setzer, Dragomir Radev, Beth Sundheim, David Day, Lisa Ferro, and Marcia Lazo. 2003b. The timebank corpus. Proceedings of Corpus Linguistics.
James Pustejovsky and Amber Stubbs. 2011. Increasing informativeness in temporal annotation. In Proceedings of the 5th Linguistic Annotation Workshop, pages 152-160.
Lianhui Qin, Aditya Gupta, Shyam Upadhyay, Luheng He, Yejin Choi, and Manaal Faruqui. 2021. Timedial: Temporal commonsense reasoning in dialog. arXiv preprint arXiv:2106.04571.
Nils Reimers and Iryna Gurevych. 2019. SentenceBERT: Sentence embeddings using Siamese BERTnetworks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982-3992, Hong Kong, China. Association for Computational Linguistics.
Hayley Ross, Jonathon Cai, and Bonan Min. 2020a. Exploring Contextualized Neural Language Models for Temporal Dependency Parsing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8548-8553, Online. Association for Computational Linguistics.
Hayley Ross, Jonathon Cai, and Bonan Min. 2020b. Exploring contextualized neural language models for temporal dependency parsing. arXiv preprint arXiv:2004.14577.
Ramit Sawhney, Puneet Mathur, Taru Jain, Akash Kumar Gautam, and Rajiv Shah. 2021. Multitask learning for emotionally analyzing sexual abuse disclosures. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4881-4892.
Ramit Sawhney, Puneet Mathur, Ayush Mangal, Piyush Khanna, Rajiv Ratn Shah, and Roger Zimmermann. 2020. Multimodal multi-task financial risk forecasting. In Proceedings of the 28th ACM international conference on multimedia, pages 456-465.
Ke Shi, Zhengyuan Liu, and Nancy F Chen. 2020. An end-to-end document-level neural discourse parser exploiting multi-granularity representations. arXiv preprint arXiv:2012.11169.
Pontus Stenetorp, Sampo Pyysalo, Goran Topić, Tomoko Ohta, Sophia Ananiadou, and Jun'ichi Tsujii. 2012. Brat: a web-based tool for nlp-assisted text annotation. In Proceedings of the Demonstrations at the 13th Conference of the European Chapter of the Association for Computational Linguistics, pages 102-107.
Jannik Strötgen and Michael Gertz. 2013. Multilingual and cross-domain temporal tagging. *Language Resources and Evaluation*, 47(2):269-298.
Siddharth Vashishtha, Adam Poliak, Yash Kumar Lal, Benjamin Van Durme, and Aaron Steven White. 2020. Temporal reasoning in natural language inference. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4070-4078, Online. Association for Computational Linguistics.
Jiarui Yao, Haoling Qiu, Bonan Min, and Nianwen Xue. 2020a. Annotating temporal dependency graphs via crowdsourcing. In EMNLP.
Jiarui Yao, Haoling Qiu, Bonan Min, and Nianwen Xue. 2020b. Annotating Temporal Dependency Graphs via Crowdsourcing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5368-5380, Online. Association for Computational Linguistics.
Liang Yao, Chengsheng Mao, and Yuan Luo. 2019. Graph convolutional networks for text classification. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 7370-7377.
Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontañón, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed. 2020a. Big bird: Transformers for longer sequences. ArXiv, abs/2007.14062.
Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Otanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. 2020b. Big bird: Transformers for longer sequences. In NeurIPS.
Yuchen Zhang and Nianwen Xue. 2018a. Neural ranking models for temporal dependency structure parsing. arXiv preprint arXiv:1809.00370.
Yuchen Zhang and Nianwen Xue. 2018b. Structured interpretation of temporal relations. arXiv preprint arXiv:1808.07599.
Yuchen Zhang and Nianwen Xue. 2019. Acquiring structured temporal representation via crowdsourcing: A feasibility study. In *SEMEVAL.
Zhenyu Zhang, Bowen Yu, Xiaobo Shu, Tingwen Liu, Hengzhu Tang, Wang Yubin, and Li Guo. 2020. Document-level relation extraction with dual-tier heterogeneous graph. In Proceedings of the 28th International Conference on Computational Linguistics, pages 1630-1641.
Xinyu Zhao, Shih-ting Lin, and Greg Durrett. 2020. Effective distant supervision for temporal relation extraction. arXiv preprint arXiv:2010.12755.
We utilize two publicly available datasets - TDT and TDG for evaluating temporal dependency parser. We also curated dataset for TDG on contract documents. We source these contract documents from a publicly available resource - ATTICUS. We repurpose the document in this dataset for our task and provide new annotations. ContractTDG dataset does not violate any privacy as these documents are already in public domain. There is no human bias involved in such documents as they are business contracts filed on the SEC website. These documents do not restrict reuse for academic purposes and any personal information was already redacted before their original release. All documents and our experiments are restricted to English language. Temporal NLI and TimeQA datasets that are publicly available for research purposes. The crowd workers are paid a fair wage. There was no sensitive data involved in the studies.
B Details on Graph Feature Extraction
B.1 Structural Graph Features
The Structural Graph  $(G_{str})$  enriches the token level features with a hierarchical textual structure formed by grouping word tokens into lists of sentences that bind together to form the text document. Prior work has shown that transductive graph learning over  $G_{str}$  can help learn the long range word-word dependencies set several sentences apart through hierarchical text modeling (Yao et al., 2019). The directed edges of the Structural Graph encode the following relationships: (1) Document-Sentence Affiliation, which connects each document-node to a sentence-node; (2) Sentence-Word Affiliation, which joins each sentence node to its constituent word nodes; (3) Sentence-Sentence Adjacency and (4) Word-Word Adjacency, which preserve sequential ordering for consecutive sentence and word nodes, respectively. For the structural graph, a sentence node embedding  $s_i$  is obtained by passing sentences through a pre-trained SentenceBERT model (Reimers and Gurevych, 2019) and the document node embedding  $D$  is calculated as the average of all sentence embeddings  $(D = \sum_{i=0}^{m} v_i)$ .
BertGCN (Lin et al., 2021) combines the advantages of both large-scale pre-training and transductive learning. We input the structural graph  $G_{str}$
to BertGCN model<sup>7</sup> where each node represents a word, a sentence or the document. BertGCN processes the input node feature matrix sequentially through a Bert model to fine-tune each node to learn local contextual representations. This is followed by passing the learned node feature matrix through two layers of graph convolution to take advantage of global influence propagation through graph edges across multi-hop nodes.
B.2 Syntactic Graph Features
Syntactic cues are useful priors for learning based NLP tasks (Kiperwasser and Ballesteros, 2018). Pre-trained transformer models can capture certain syntactic information implicitly (Hewitt and Manning, 2019) but Jawahar et al. (2019) showed that BERT needs to be trained with deeper layers for handling harder cases involving long-distance dependency information. Moreover, past studies have pointed to the existence of multi-hop coreferring expressions in document-level text due to anaphora and cataphora (Joshi et al., 2020).
$G_{syn}$  is made of separate nodes to represent each constituent word  $w_{i}$  in the document. For each document, there is also a set of co-reference clusters  $\{\mathbb{C}_1,\mathbb{C}_2,\dots ,\mathbb{C}_u\}$  referring to the same entities in the graph. We define four types of directed edges in  $G_{syn}$  as described below where  $\xi$  denotes the set of syntactic dependency arcs inside sentences,  $S_{w_i}^r$  denotes root of the sentence in which  $w_{i}$  belongs, and  $S_{w_i}\rightarrow S_{w_j}$  represents whether sentences containing words  $w_{i}$  and  $w_{j}$  are adjacent.
$$
\varepsilon_ {s y n} (i, j) = \left\{\begin{array}{l l}\text {d e p e n d e n c y}&\text {i f} \left(w _ {i}, w _ {j}\right) \in \xi\\\text {r e v e r s i o n}&\text {i f} \left(w _ {j}, w _ {i}\right) \in \xi\\\text {c o r e f e r e n c e}&\text {i f} w _ {i}, w _ {j} \in \mathbb {C} _ {u}\\\text {s e l f - l o o p}&\text {i f} i = = j\\\text {r o o t - a d j a c e n c y}&\text {i f} w _ {i} = = S _ {w _ {i}} ^ {r},\\&\&w _ {j} = = S _ {w _ {j}} ^ {r},\\&\&S _ {w _ {i}} \rightarrow S _ {w _ {j}}\end{array}\right. \tag {3}
$$
The first two edge types are introduced to allow information flow along and against syntactic arcs between intra-sentential dependency relations to enrich contextually learned embeddings of each word. We connect parse tree roots of adjacent sentences to encode document level long-range syntactic relatedness between sentences. We add an undirected edge between word nodes if both belong to the same co-reference cluster. Inspired by (Kipf and Welling, 2016), self-loop edges are added for better message passing iterations.  $G_{syn}$  is instantiated
7Implementation Used: https://github.com/ZeroRin/BertGCN
as a gated variant of Weighted Relational Graph Convolutional Network (WR-GCN) (Zhang et al., 2020) with  $k$ -layers. WR-GCN can be able to model diverse relations in a heterogeneous graph by treating different types of edges with unequal weights assigned during message passing.
B.3 Semantic Graph Features
Semantic Role Labeling (SRL) parses text sequences to recognize the predicate-argument structure in the sentence to answer who did what and when. Anchoring verb events to their temporal argument spans extracted from semantic parsing helps infer event relationships with their associated time expressions. This can be complemented by discourse features in the form of RST connections can help leverage long-range document-level interactions between phrase units (Bhatia et al., 2015) and identify background foreground events(Aldawsari et al., 2020) and improve temporal relationship parsing (Mathur et al., 2021b). We utilize Document-level Rhetorical Structure Theory (RST) parser (Shi et al., 2020) to organize contiguous semantic text spans of a document into a hierarchical dependency structure labeled with their rhetorical relations.
$G_{sem}$  consists of individual nodes for each constituent word  $w_{i}$  in the document. Discourse units and temporal arguments may span several word tokens  $\{w_{1},w_{2},\dots w_{k}\}$ . We add two types of directed edge connections between - (1) event verb predicate - temporal argument edge  $(\varepsilon_{t})$  such that  $(w_{e}\rightarrow \{w_{1},w_{2},\dots w_{k}\} \in \varepsilon_{t})$ ; (2) Rhetorical pair edges  $(\varepsilon_{d})$  labelled by the type of the rhetorical relation  $(\{w_{1},w_{2},\dots w_{i}\} \to \{w_{1},w_{2},\dots w_{j}\} \in \varepsilon_{d})$ .
$$
\varepsilon = \left\{\begin{array}{l}w _ {e} \rightarrow \left\{w _ {e}, \dots w _ {k} \right\} \in \varepsilon_ {t}\\\left\{w _ {1}, \dots w _ {i} \right\}\rightarrow \left\{w _ {1}, \dots w _ {j} \right\} \in \varepsilon_ {d}\end{array}\right. \tag {4}
$$
The nature of edge connections in  $G_{sem}$  extends beyond pairwise interactions as each edge may connect to one or more word nodes. Hence, we formulate the semantic graph as a hypergraph (Feng et al., 2019) where an edge can join an arbitrary number of vertices. We construct  $G_{sem} = (\nu, \varepsilon, \mathbf{W})$  where  $\nu$  is the set of all word nodes  $w_i$ , and  $\varepsilon$  is the subset of hyperedges such that  $\varepsilon = \varepsilon_t \cup \varepsilon_d$ . Each hyperedge  $e$  is assigned a positive weight corresponding to the type of edge relation and is stored in a diagonal matrix  $\mathbf{W} \in \Re^{|\varepsilon| x|\varepsilon|}$ . The semantic graph is learned using hypergraph convolution layers (Bai et al., 2021b) to obtain discriminative node
embeddings for each word node.
C Training Setup
Hyperparameter: Hyper-parameters for DocTime were tuned on the respective validation set to find the best configurations for different datasets. We summarize the range of our model's hyper parameters such as: number of hidden layers in WR-GCN/BERT-GCN/HyperGraphGCN  $\{1,2,3\}$ , size of hidden layers in WR-GCN/BERT-GCN/HyperGraphGCN  $\{64,128,256,512\}$ , BERT embedding size (768), dropout  $\delta \in \{0.2,0.3,0.4,0.5,0.6\}$ , learning rate  $\lambda \in \{1e - 5,1e - 4,1e - 3,1e - 2,1e - 1\}$ , weight decay  $\omega \in \{1e - 6,1e - 5,1e - 4,1e - 3\}$ , batch size  $b\in \{16,32,64\}$  and epochs ( $\leq 100$ ),  $\epsilon$ -sparsity  $\in [0,1]$ , IDGL smoothness ratio=0.5, IDGL sparsity ratio=0.5, IDGL connectivity ratio=0.5, size of hidden layers in Graph U-net  $\{64,128,256,512\}$ .
Loss Function and Inference:
Time-Transformers are trained using Cross Entropy loss with Adam optimizer. Across both TempNLI and TimeQA datasets, we found the best results correspond with the use of Adam optimiser set with default values  $\beta_{1} = 0.9$ ,  $\beta_{2} = 0.999$ ,  $\epsilon = 1e - 8$ , weight-decay of  $5e - 4$  and an initial learning rate of 0.001.
DocTime uses cross entropy loss for structure prediction. For structure+relation classification, it uses the path prediction loss as defined in Methodology.
Computing Infrastructure: DocTime and Time-Tranformers are written in PyTorch library and were trained on 4 and 6 Nvidia GeForce RTX 2080 GPU, respectively. Average Runtime: DocTime takes a maximum of approximately 5 hrs to train once on TDG datasets. Time-BERT, Time-RoBERTa take 3 hrs to fine-tune on TempNLI. Time-BigBird, Time-FiD takes 8,12 hours to fine-tune, respectively.
Dataset Access
Links to download TDT dataset: https://github.com/yuchenz/crowdsourced_EN_TDT Corpus
Link to download TDG dataset: https://github.com/Jryao/temporal Dependency_graphscrowdsourcing
Link to download Temporal NLI dataset: https://github.com/sidsvash26/temporal_nli
Link to download TimeQA dataset: https://
Corpus	Model	Structure + Relation (F1)
te,te	e,te	e,e	full
TD-Graphs	Heuristic	0.82	0.58	0.34	0.51
Neural Ranking Parser (Zhang and Xue, 2018a)	0.93	0.66	0.58	0.66
BERT Ranking Parser (Ross et al., 2020b)	0.93	0.74	0.58	0.71
DocTime	0.96	0.75	0.72	0.77
Contract-TDG	Heuristic	0.45	0.36	0.18	0.33
Neural Ranking Parser (Zhang and Xue, 2018a)	0.57	0.45	0.29	0.48
BERT Ranking Parser (Ross et al., 2020b)	0.70	0.54	0.33	0.61
DocTime	0.75	0.56	0.39	0.64
Table 7: Performance (F1 score) of DocTime across timemax, event-timex and event-event pairs for dependency structure+relation prediction on TDG and ContractTDG datasets. DocTime outperforms all baselines on every setting.
github.com/wenhuchen/Time-Sensitive-QA
Table 8 show the Training hyperparameters of DocTime for TDT, TDG, ContractTDG datasets.
E More Results
Performance across different relation types: We analyze the benefits of DocTime for different types of relations in document-level TDG datasets in Table 7. We report F1 scores for structure+relation prediction for timex-timex, event-timex and event-event pairs. We observe a relatively smaller performance gap between the BERT Ranking parser and DocTime for event-timex relations. However, DocTime shows relatively stronger performance for event-event relations. This phenomenon can be attributed to the fact that both datasets tend to have event-event links between event pairs that are on an average closer in word distance, whereas a higher ratio of event-timex and timex-timex pairs are several sentences apart. DocTime can integrate long-range interdependencies between entity pairs that are several sentences (or paragraphs in Contract TDG) apart.
Dataset
Hyperparameters	TDT	TDG	Contract
Dropout Ratio	0.5	0.5	0.5
Optimizer	Adam	Adam	Adam
Input Dimension (Structural Graph)	(n,768)	(n,768)	(n,768)
Input Dimension (Syntactic Graph)	(n,768)	(n,768)	(n,768)
Input Dimension (Semantic Graph)	(n,768)	(n,768)	(n,768)
Hidden Dimension (WR-GCN)	256	256	64
Number of hidden layers (WR-GCN)	2	2	2
Hidden Dimension (BERT-GCN)	256	256	64
Number of hidden layers (BERT-GCN)	1	1	1
Hidden Dimension (HyperGCN)	256	256	64
Number of hidden layers (HyperGCN)	2	2	2
Epochs	20	20	20
Batch Size	8	8	16
Learning Rate	2e-5	2e-5	2e-5
Activation Function of Linear layers	ReLU	ReLU	ReLU
Table 8: Hyperparameters Details: Training hyperparameters of DocTime for TDT, TDG, ContractTDG