SlowGuess/NAACL_2022_OCR huggingface dataset:
From Table 3, we see that our approach achieves better performance on event detection than baseline methods though they are based on the same BERT encoder. This is possibly because, during joint training, our approach leverages the dependency parsing trees, which improves the contextual representations of the BERT encoder. In Table 4, unlike other models which are based on larger contextualized embeddings such as RoBERTa, our approach with BERT base achieves comparable performance, and further surpasses the state-of-the-art baseline methods using BERT-large embeddings, which demonstrate the effectiveness of our Syntax-guided Graph Transformer network.
Some studies (Ning et al., 2019; Han et al., 2019b; Wang et al., 2020a; Zhou et al., 2020) focus on resolving the inconsistency in terms of the symmetry and transitivity of the temporal relations. For example, if event A and event B are predicted as Before, event B and event C are predicted as Before, then if event A and event C are predicted as Vague or After, it will be considered as inconsistent. How-
Table 4: Comparison of various approaches on temporal relation classification with gold event mentions as input.
Model	Original Test	Contrast	Consistency
CogCompTime2.0 (Ning et al., 2019)	73.2	63.3	40.6
Our Approach	77.0	64.8	49.8
Table 5: Evaluation on the contrast set of MATRES. Original Test indicates the accuracy on 100 examples sampled from the original MATRES test set following (Gardner et al., 2020). Contrast shows the accuracy score on 401 examples perturbed from the original 100 examples. Consistency is defined as the percentage of the original 100 examples for which the model's predictions of the perturbed examples are all correct in the contrast set.
ever, our approach shows consistent predictions with few inconsistent cases when Simultaneous relation is involved. This analysis also demonstrates that our approach can correctly capture the temporal cues between two event mentions.
We also examine the correctness and robustness of our approach on a contrast set of MATRES (Gardner et al., 2020), which is created with small manual perturbation based on the original test set of MATRES in a meaningful way, such as rephrasing the sentence or simply changing a word of the sentence to alter the relation type. The contrast set
Figure 3: Comparison of the predictions from BERT, BERT-GT and our approach.
provides a local view of a model's decision boundary, thus it can be used to more accurately evaluate a model's true linguistic capabilities. Table 5 shows that our approach significantly outperforms the baseline model on both the original test set and the corresponding contrast set. The contrast consistency in Table 5 also indicates how well a model's decision boundary aligns with the actual decision boundary of the test instances, based on which we can see that by explicitly capturing temporal cues, our approach is more accurate and robust than the baseline method.
Ablation Study We further conduct ablation studies to compare the performance of our approach with two ablated versions of our method: (1) BERT with Graph Transformer (BERT-GT), for which we remove the syntaxic-guided attention and only rely on the standard multi-head self-attention to obtain graph-based contextual representations of two event mentions and then predict their relation; (2) BERT, where we further remove the Graph Transformer, and only use the pre-trained BERT language model to encode the sentence and predict the temporal relationship of two event mentions based on their contextual representations.
Ablation	F-score (%)	Gain (%)
BERT-SGT	79.3	0
BERT-GT	77.5	-2.0
BERT	75.5	-3.8
Table 6: Ablation study on MATRES. We use BERT base as the comparison basis.
Table 6 also shows that by adding Graph Transformer, BERT-GT achieves  $2.0\%$  absolute F-score improvement over the BERT baseline model, demonstrating the benefit of dependency parsing trees to temporal relation prediction. By further adding the new syntax-guided attention into Graph Transformer, the absolute improvement on F-score  $(1.8\%)$  shows the effectiveness of our new Syntax-guided Graph Transformer and the importance of
capturing temporal cues from the connection of two event mentions as well as their surround contexts.
Figure 3 shows two examples as qualitative analysis. In S1, BERT mistakenly predicts the temporal relation as Before probably because it's confused by the context word Before. However, by incorporating the dependency graph, especially the triples {worked, prep, Before}, {Before, pcomp, retiring} and the path between the two event mentions, worked  $\rightarrow$  prep  $\rightarrow$  Before  $\rightarrow$  pcomp  $\rightarrow$  retiring, both BERT-GT and our approach correctly determine the relation as After. In S2, both BERT and BERT-GT mistakenly predict the temporal relation as Before as the context between the two event mentions is very wide and complicated, and these two event mentions are not close within the dependency graph. However, by explicitly considering and understanding the connection between the two event mentions,  $sought_{e_1} \rightarrow on \rightarrow Marmara \rightarrow was \rightarrow part \rightarrow Flotilla \rightarrow sought_{e_2}$ , our approach correctly predicts the temporal relation between the two event mentions.
3.3 SGT on Temporal Cues
To analyze the source of temporal cues for relation prediction, we randomly sample 100 correct event relation predictions given gold event mentions from MATRES and select the triple that has the highest temporal attention weight from the last layer of the Syntax-guided Graph Transformer network as a temporal cue candidate. We manually evaluate the validity of each temporal cue candidate, and further analyze if the cue is from the dependency path between two event mentions, their surrounding context, or both. Our analysis shows that about  $64\%$  of the temporal cues are valid,  $37\%$  of them come from the dependency path,  $17\%$  are from local context, and the remaining  $10\%$  are from both. This verifies our initial observation that most of the temporal cues are from the dependency path between two event mentions as well as their surrounding context. It also demonstrates the
effectiveness of our new syntax-guided attention mechanism.
3.4 Impact of Wide Context
We further illustrate the impact of context width to both baseline model and our approach. For fair comparison, we use three context width category, [context length  $< 10$ ,  $10 <$  context length  $< 20$ , context length  $>20$ ]. As we can see in Fig. 4, the first category has 267 pairs, the second category has 343 pairs and the third category has 817 pairs. From our results, we observe that the BERT baseline cannot predict the temporal relation of two event mentions with wide context but rather working well when the event mentions are close to each other. Our model overall performs slightly worse in the second category but in general is very good at predicting the temporal relationship for the event mentions with short and context width. This also proves the benefit of syntactic parsing trees to the prediction of temporal relationship. For the second category where the context length is within [10, 20], the performance of our approach slightly drops due to two reasons: (1) the training samples within this range are not as sufficient as the other two categories; (2) for most event pairs from this category, their dependency path is very long and there is no explicit temporal indicative features within their context or dependency path, making it more difficult for the model to predict their temporal relationship.
Figure 4: Context width analysis on TB-DENSE. The X axis shows the number of tokens between two events mentions. The left Y axis shows the data distribution of each width category indicating with blue bars. The right Y axis denotes the micro F-score for each width category.
3.5 Remaining Errors
We randomly sample 100 classification errors from the output of our approach and categorize them into four categories. As Figure 5 shows, the first category is due to the complex or ambiguous context (54% of the total errors). The second category is due to the complicated subordinate clause structure, especially the clauses that are related to quote or reported speech, e.g., S2 in Figure 5. The third error category is that our approach cannot correctly differentiate the actual events from the hypothetical and intentional events, while in most cases, the temporal relation among hypothetical and intentional events is annotated as Vague. The last category is due to the lack of sufficient annotation. We observe that none of the Simultaneous relation can be correctly predicted for MATRES dataset as the percentage of Simultaneous (3.7%) is much lower than other relation types. In TB-DENSE dataset, labels are even more imbalanced as the percentage of Vague relation is over 50% while the percentage of Includes, Is_Included and Simultaneous are all less than 4%.
Early studies on temporal relation extraction mainly model it as a pairwise classification problem (Mani et al., 2006; Verhagen et al., 2007; Verhagen and Pustejovsky, 2008; Verhagen et al., 2010; Bethard et al., 2016; MacAvaney et al., 2017) and rely on hand-crafted features and rules (Verhagen and Pustejovsky, 2008; Bethard et al., 2007) to extract temporal event relations. Recently, deep neural networks (Diligach et al., 2017; Tourille et al., 2017) and large-scale pre-trained language models (Han et al., 2019a, 2021; Wang et al., 2020a; Zhou et al., 2020) are further employed and show state-of-the-art performance.
Similar to our approach, several studies (Ling and Weld, 2010; Nikfarjam et al., 2013; Mirza and Tonelli, 2016; Meng et al., 2017; Cheng and Miyao, 2017; Huang et al., 2017) also explored syntactic path between two events for temporal relation extraction. Different from previous work, our approach considers three important sources of temporal cues: local context, denoting the neighbors of each event node within the dependency graph; connection of two event mentions, which is based on the dependency path between two event mentions; and rich semantics of concepts and dependency relations, for example, the dependency
Error Category (Percent)	Example
Complex Context (54%)	S1: "This is not a Lehman , " he (e1: said) to the disastrous chain reaction (e2: touched) off by the collapse of Lehman brothers in 2008 . (After)
Subordinate Clause (22%)	S2: "We were pleased that England and New Zealand knew about it, and we (e1: thought) that's where it would stop." He also (e2: talked) about his " second job " as the group's cameraman. (Vague)
Hypothetical Events and Intentional Events (18%)	S3: The day before Raymond Roth was (e1: pulled) over, his wife, Ivana, showed authorities emails she had discovered that (e2: appeared) to detail a plan between him and his son to fake his death. (Vague)
Imbalanced Labels (6%)	S4: Microsoft (e1: said) it has identified three companies for the china program to (e2: run) through June . (Simultaneous)
Figure 5: Types of remaining errors
relation nmod between two event mentions usually indicates a Before relationship. All these indicative features are automatically selected and aggregated with the multi-head self-attention and our new syntax-guided attention mechanism.
Our work is also related to the variants of Graph Neural Networks (GNN) (Kipf and Welling, 2016; Velicković et al., 2018; Zhou et al., 2018), especially Graph Transformer (Yun et al., 2019; Chen et al., 2019; Hu et al., 2020; Wang et al., 2020b). Different from previous GNNs which aim to capture the context from neighbors of each node within the graph, in our task, we aim to select and capture the most meaningful temporal cues for two event mentions from their connections within the graph as well as their surrounding contexts.
Temporal relationship between events is important for understanding stories described in natural language text, and a main challenge is how to discover and make use of the connection between two event mentions, especially when the event pair is far apart in text. This paper proposes a novel Syntax-guided Graph Transformer (SGT) that represents the connection between an event pair via additional attention heads over dependency parsing trees. Experiments on benchmarking datasets, MATRES, TB-DENSE, and a contrast set of MATRES, show that our approach significantly outperforms previous state-of-the-art methods in a variety of settings, including event detection, temporal relation classification (where events are given), and temporal relation extraction (where events are predicted). In the future, we will investigate the potential of this approach to other relation extraction tasks.
We thank the anonymous reviewers and area chair for their valuable time and constructive comments. We also thank the support from the Amazon Research Awards.
S. Bethard, J. H. Martin, and S. Klingenstein. 2007. Timelines from text: Identification of syntactic temporal relations. In International Conference on Semantic Computing (ICSC 2007), pages 11-18.
Steven Bethard, Guergana Savova, Wei-Te Chen, Leon Derczynski, James Pustejovsky, and Marc Verhagen. 2016. Semeval-2016 task 12: Clinical tempeval. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), pages 1052-1062.
Taylor Cassidy, Bill McDowell, Nathanel Chambers, and Steven Bethard. 2014. An annotation framework for dense event ordering. Technical report, CARNEGIE-MELLON UNIV PITTSBURGH PA.
Nathanael Chambers. 2013. NavyTime: Event and time ordering from raw text. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013), pages 73-77, Atlanta, Georgia, USA. Association for Computational Linguistics.
Nathanael Chambers, Taylor Cassidy, Bill McDowell, and Steven Bethard. 2014. Dense event ordering with a multi-pass architecture. Transactions of the Association for Computational Linguistics, 2:273-284.
Benson Chen, Regina Barzilay, and Tommi Jaakkola. 2019. Path-augmented graph transformer network. arXiv preprint arXiv:1905.12712.
Muhao Chen, Hongming Zhang, Qiang Ning, Manling Li, Heng Ji, Kathleen McKeown, and Dan Roth. 2021.
Event-centric natural language processing. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: Tutorial Abstracts, pages 6-14, Online. Association for Computational Linguistics.
Fei Cheng and Yusuke Miyao. 2017. Classifying temporal relations by bidirectional LSTM over dependency paths. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 1-6.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
Dmitriy Dligach, Timothy Miller, Chen Lin, Steven Bethard, and Guergana Savova. 2017. Neural temporal relation extraction. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 746-751.
Matt Gardner, Yoav Artzi, Victoria Basmov, Jonathan Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi, Dheeru Dua, Yanai Elazar, Ananth Gottumukkala, Nitish Gupta, Hannaneh Hajishirzi, Gabriel Ilharco, Daniel Khashabi, Kevin Lin, Jiangming Liu, Nelson F. Liu, Phoebe Mulcaire, Qiang Ning, Sameer Singh, Noah A. Smith, Sanjay Subramanian, Reut Tsarfaty, Eric Wallace, Ally Zhang, and Ben Zhou. 2020. Evaluating models' local decision boundaries via contrast sets. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1307-1323, Online. Association for Computational Linguistics.
Rujun Han, I-Hung Hsu, Mu Yang, Aram Galstyan, Ralph Weischedel, and Nanyun Peng. 2019a. Deep structured neural network for event temporal relation extraction. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 666-106, Hong Kong, China. Association for Computational Linguistics.
Rujun Han, Qiang Ning, and Nanyun Peng. 2019b. Joint event and temporal relation extraction with shared representations and structured prediction. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 434-444, Hong Kong, China. Association for Computational Linguistics.
Rujun Han, Xiang Ren, and Nanyun Peng. 2021. ECONET: Effective continual pretraining of language models for event temporal reasoning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5367-5380, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Sanda Harabagiu and Cosmin Adrian Bejan. 2005. Question answering based on temporal inference. In Proceedings of the AAAI-2005 workshop on inference for textual question answering, pages 27-34.
Ziniu Hu, Yuxiao Dong, Kuansan Wang, and Yizhou Sun. 2020. Heterogeneous graph transformer. In Proceedings of The Web Conference 2020, pages 2704-2710.
Lifu Huang, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2019. Cosmos qa: Machine reading comprehension with contextual commonsense reasoning. arXiv preprint arXiv:1909.00277.
Lifu Huang and Heng Ji. 2020. Semi-supervised new event type induction and event detection. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 718-724.
Lifu Huang, Avirup Sil, Heng Ji, and Radu Florian. 2017. Improving slot filling performance with attentive neural networks on dependency structures. arXiv preprint arXiv:1707.01075.
Yexi Jiang, Chang-Shing Perng, and Tao Li. 2011. Natural event summarization. In Proceedings of the 20th ACM international conference on Information and knowledge management, pages 765-774.
Thomas N Kipf and Max Welling. 2016. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907.
Manling Li, Sha Li, Zhenhailong Wang, Lifu Huang, Kyunghyun Cho, Heng Ji, Jiawei Han, and Clare Voss. 2021. The future is not one-dimensional: Complex event schema induction by graph modeling for event prediction. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5203-5215.
Li Lin, Yixin Cao, Lifu Huang, Shuang Li, Xuming Hu, Lijie Wen, and Jianmin Wang. 2022. Inferring commonsense explanations as prompts for future event generation. arXiv preprint arXiv:2201.07099.
Ying Lin, Heng Ji, Fei Huang, and Lingfei Wu. 2020. A joint neural model for information extraction with global features. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7999-8009.
Xiao Ling and Daniel Weld. 2010. Temporal information extraction. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 24.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Sean MacAvaney, Arman Cohan, and Nazli Goharian. 2017. Guir at semeval-2017 task 12: a framework for cross-domain clinical temporal information extraction. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 1024–1029.
Inderjeet Mani, Marc Verhagen, Ben Wellner, Chung-min Lee, and James Pustejovsky. 2006. Machine learning of temporal relations. In Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics, pages 753-760.
Yuanliang Meng, Anna Rumshisky, and Alexey Romanov. 2017. Temporal information extraction for question answering using syntactic dependencies in an LSTM-based architecture. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 887-896, Copenhagen, Denmark. Association for Computational Linguistics.
Paramita Mirza and Sara Tonelli. 2016. Catena: Causal and temporal relation extraction from natural language texts. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 64-75.
Jun Ping Ng, Yan Chen, Min-Yen Kan, and Zhoujun Li. 2014. Exploiting timelines to enhance multi-document summarization. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 923-933.
Azadeh Nikfarjam, Ehsan Emadzadeh, and Graciela Gonzalez. 2013. Towards generating a patient's timeline: extracting temporal relationships from clinical notes. Journal of biomedical informatics, 46:S40-S47.
Qiang Ning, Zhili Feng, and Dan Roth. 2017. A structured learning approach to temporal relation extraction. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1027-1037, Copenhagen, Denmark. Association for Computational Linguistics.
Qiang Ning, Sanjay Subramanian, and Dan Roth. 2019. An improved neural baseline for temporal relation extraction. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 6203-6209, Hong Kong, China. Association for Computational Linguistics.
Qiang Ning, Hao Wu, Rujun Han, Nanyun Peng, Matt Gardner, and Dan Roth. 2020. TORQUE: A reading comprehension dataset of temporal ordering questions. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1158-1172, Online. Association for Computational Linguistics.
Qiang Ning, Hao Wu, and Dan Roth. 2018. A multi-axis annotation scheme for event temporal relations. In ACL.
Nanyun Peng, Hoifung Poon, Chris Quirk, Kristina Toutanova, and Wen-tau Yih. 2017. Cross-sentence n-ary relation extraction with graph lstms. Transactions of the Association for Computational Linguistics, 5:101-115.
Alan Ritter, Colin Cherry, and William B Dolan. 2010. Unsupervised modeling of twitter conversations. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 172-180.
Yajing Sun, Yong Shan, Chengguang Tang, Yue Hu, Yinpei Dai, Jing Yu, Jian Sun, Fei Huang, and Luo Si. 2021. Unsupervised learning of deterministic dialogue structure with edge-enhanced graph autoencoder. In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021, pages 13869-13877. AAAI Press.
Yawei Sun, Gong Cheng, and Yuzhong Qu. 2018. Reading comprehension with graph-based temporal-casual reasoning. In Proceedings of the 27th International Conference on Computational Linguistics, pages 806-817.
Julien Tourille, Olivier Ferret, Aurelie Neveol, and Xavier Tannier. 2017. Neural architecture for temporal relation extraction: A bi-lstm approach for detecting narrative containers. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 224-230.
Siddharth Vashishtha, Benjamin Van Durme, and Aaron Steven White. 2019. Fine-grained temporal relation extraction. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2906-2919, Florence, Italy. Association for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. arXiv preprint arXiv:1706.03762.
Petar Velicković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. 2018. Graph Attention Networks. International Conference on Learning Representations. Accepted as poster.
Marc Verhagen, Robert Gaizauskas, Frank Schilder, Mark Hepple, Graham Katz, and James Pustejovsky. 2007. Semeval-2007 task 15: Tempeval temporal relation identification. In Proceedings of the