SlowGuess/NAACL_2022_OCR huggingface dataset:
Evaluation Metrics Following Ning et al. (2020) $^3$ , we report three metrics in our experiment, including standard macro F1 and Exact Match (EM) for question answering and consistency score(C). There are multiple annotations for each passage-question pair, which might not always be consistent with each other. We follow the official implementation. Specifically, for each sample, a model's prediction is evaluated according to all annotations, where the largest score is selected and aggregated as the final result.
We experiment four pre-trained language models as our contextual encoder, i.e., the base and large model of BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019). The embedding size  $d$  is set to 64,  $d'$  in Eq (6) and Eq (7) is set to 64. The threshold  $\tau$  for inference is set to 0.5. In model training, the batch size is set to
16, the dropout rate is set to 0.5. The combination weight  $w_{qa}$ ,  $w_{att}$  and  $w_{con}$  in Eq. (8) is set to 1.0, 0.3, and 1.0, respectively. We search the learning rate  $lr$ , with grid searching within 3 trials in  $lr \in \{0.9 \times 10^{-5}, 1.0 \times 10^{-5}, 1.1 \times 10^{-5}\}$  for the base and large model of RoBERTa, and  $lr \in \{4.0 \times 10^{-5}, 5.0 \times 10^{-5}, 6.0 \times 10^{-5}\}$  for the base and large model of BERT. The implementation is based on Python and trained on a Tesla V100 GPU with Adam optimizer for approximately three hours (base model with approximately 110M parameters) and ten hours (large model with approximately 340M parameters). We get the averaged result of three trials for each setting, choose the model with the highest F1 score on the development set, and report the performance on the test set derived from the official online test<sup>4</sup>.
Deriving Attention Annotation The relation annotation  $Q_{r}$  for question  $Q$  is derived as follows. First, we compile a dictionary for temporal relations, such as before, after, etc. Please refer to Appendix A.1 for the complete list. Then  $Q_{r}$  is constructed with those words in  $Q$  that hit the dictionary. The event annotation  $Q_{c}$  is mainly derived according to the passage  $P$ . Particularly, we assume the mentioned event list  $E$  in  $P$  is known. If a word of  $Q$  matches an event in  $E$ , it is included in  $Q_{c}$ . Otherwise, if no words of  $Q$  hit  $E$ , we rely on the relation annotation. Suppose the last relation word is in position  $k$ , then  $Q_{k + 1\dots n}$  is set as  $Q_{c}$ .
We compare our approach with the baseline (Ning et al., 2020), which takes a passage and the corresponding question as input and applies a one-layer perception on the embedding of each token to predict whether it is the answer of the question or not. The comparison results with four different contextual encoders are shown in Table 1. The table shows that our proposed approach outperforms the baseline on nearly all evaluation metrics. Our model achieves state-of-the-art results with the RoBERTa-large encoder, increasing the F1 score by  $1.8\%$  and  $0.9\%$  for the dev and test set, respectively. We can see a huge increase for the consistency score (C) on the test set from  $34.5\%$  to  $38.1\%$ . Using other pretrain language models like BERT-base, our model also improves the performance compared to the baseline approach, by  $2.6\%$ ,  $3.2\%$ ,  $2.5\%$  in terms of F1, EM, and C score, respectively. Although
$^{2}$ https://github.com/qiangning/TORQUE-dataset
<sup>3</sup>https://github.com/rujunhan/TORQUE
4 https://leaderboard.allenai.org/torque/submissions/public
F1	Dev EM	C	F1	Test EM	C
BERT-base
baseline†	67.6	39.6	24.3	67.2	39.8	23.6
Ours	70.5	44.6	26.2	69.8	43.0	26.1
Δ	+2.9	+5.0	+1.9	+2.6	+3.2	+2.5
BERT-large
Baseline†	72.8	46.0	30.7	71.9	45.9	29.1
Ours	73.5	46.5	31.8	72.6	45.1	30.1
Δ	+0.7	+0.5	+1.1	+0.7	-0.8	+1.0
RoBERTa-base
Baseline†	72.2	44.5	28.7	72.6	45.7	29.9
Ours	73.3	47.0	32.5	73.5	46.8	31.5
Δ	+1.1	+3.5	+3.8	+0.9	+1.1	+1.6
RoBERTa-large
Baseline†	75.7	50.4	36.0	75.2	51.1	34.5
Ours	77.5	52.2	37.5	76.1	51.0	38.1
Δ	+1.8	+1.8	+1.5	+0.9	-0.1	+3.6
Human	-	-	-	95.3	84.5	82.5
there is still a large gap towards the human performance, our model takes a large step compared to the baseline approach, verifying the effectiveness of the proposed approach.
Table 1: Comparison of our approach and the baseline on the TORQUE Dataset. † denotes published results (Ning et al., 2020).
Models	F1	EM	C
OUR MODEL	76.1	51.0	38.1
-con	75.8 (-0.3)	49.8 (-1.2)	37.0 (-1.1)
-con -att	75.6 (-0.5)	50.8 (-0.2)	36.6 (-1.5)
- \(w_e\)	75.8 (-0.3)	50.6 (-0.4)	37.6 (-0.5)
-all	74.8 (-1.3)	49.7 (-1.3)	34.0 (-4.1)
We conduct an ablation study to illustrate the effectiveness of each loss in our approach. As shown in Table 2, removing the contrastive loss will lead to a  $1.1\%$  drop on consistency value. When we remove both the contrastive and attention loss for question understanding and use mean pooling over the contextual embedding of the whole question token sequence, the macro F1 score and the consistency score decrease by  $0.5\%$  and  $1.5\%$ , respectively, showing that precise question understanding plays a critical role for TRC. Also, we remove weight  $w_{e}$  in the answer prediction loss in Eq. (9), which results in a  $0.3\%$  drop in terms of the F1 score. When all auxiliary loss is removed, which
is basically the same as the baseline model with our own implementation, it leads to a huge gap of  $1.3\%$ ,  $1.3\%$ ,  $4.1\%$  on macro F1, exactly match and Consistency score, respectively. The results of the ablation study indicate that each element of our proposed model is critical for temporal relation understanding.
4.5 Question Representation Analysis
Table 2: Ablation study on the test set of TORQUE. RoBERTa-large is used as contextual encoder.
Models	F1	EM	C
w contrastive loss
attention-based	76.1	51.0	38.1
rule-based	75.8 (-0.3)	50.6 (-0.4)	37.6 (-0.5)
w/o contrastive loss
attention-based	75.8	49.8	37.0
rule-based	75.6 (-0.2)	48.9 (-0.9)	36.3 (-0.7)
Table 3: Comparison of attention-based and rule-based question representation learning. RoBERTa-large is used as contextual encoder.
As discussed in Section 3.2, a straightforward solution for question understanding is to decompose a temporal ordering question into two parts directly. This section compares our attention-based approach with the hard question decomposition, which obtains the two question vectors  $\mathbf{h}_{\mathbf{r}}$  and  $\mathbf{h}_{\mathbf{c}}$  by conducting mean pooling over embeddings of tokens in  $Q_{r}$  and  $Q_{c}$  respectively. The comparison results are shown in Table 3. We can see that although the rule-based approach achieves relatively good accuracy, it still underperforms our attention-based approach. For example, when no contrastive loss is employed, the EM score drops by  $0.9\%$  when replacing the attention-based representation with the rule-based one. The possible reason is that the rule-based decomposition cannot handle all questions perfectly, and errors in the decomposition will be propagated to downstream modules. For example, "What could have happened while the announcement was made but didn't?". "but didn't" is a crucial negate in the temporal relation, but the rule-based method might miss it.
Figure 6 shows predicted answers of our model and the baseline for several questions. For the first passage, Questions 1, 2, and 3 inquire about the "happened after" temporal relation, but with subtle differences. Q1 is the most common form, which can be answered correctly by both the baseline and our proposed approach. Meanwhile, the baseline model can not capture the negation information
Paragraph: “This decision came after the failure of the dialogue to form a national unity government with the Hamas movement and is the result of the ongoing political and economic siege against the Palestinian people,” he added.
Question & Answers
Q 1: What happened after the dialogue began?
Q 2: What has not happened after the dialogue began?
Q 3: According to the speaker, what began happening after the dialogue began?	Baseline
decision, failure
No answer
decision, failure	Ours
decision, failure
form
failure

Question & Answers
Q 1: What might have started after the reports?
Q 2: What might have started before the reports?
Q 3: What started before the reports?	Baseline
divert, turbulence
increase, turbulence
increase, turbulence	Ours
divert
increase
turbulence
Figure 6: Case study of our approach and the baseline model. Correct answers are marked in blue. Incorrect ones are marked in red. Candidate events in passages are underlined. Both the baseline and our approach use RoBERTa-large as encoder.
in Q2 and fails to predict the correct answer. In Q3 "happened after" is constrained by the word begin, which confuses the baseline model and leads to partially correct answers. In contrast, our proposed approach can capture these subtle but critical differences and thus makes correct predictions.
For the second passage, our proposed model performs better for all three questions of different temporal types. Q1 and Q2 are variants of uncertain relations, which query about two opposite temporal relations "started after" and "started before". The word "might" brings uncertainty for the concerned temporal relation, which confuses the baseline model, leading to the wrong prediction for the candidate answer "turbulence" for both questions. Q3 queries about a popular temporal relation, and our model can precisely capture the difference between it and two other ones and predict that the candidate event "increase" does not meet its requirement since it comes from a controversial report.
We randomly sample 100 wrongly predicted question-passage pairs from the validation set, which can be summarized into three categories.
Multi-round Reasoning Sometimes one needs to perform multi-round reasoning to infer the relation between two events, for example, given the passage "Roughly 40 minutes after the operation began, jubilant soldiers appeared on the rooftop of the residence, flashing the V victory sign. Then Fujimori, who ordered the operation, arrived to tour the residence and embraced the freed hostages."
the temporal ordering between “ordered” and “the jubilant soldiers appeared on the rooftop” is inferred by multi-step reasoning. That is, “ordered” happened before “operation began”, and “operation began” happened before “solder appeared”, and thus “ordered” happened before “appeared”. An advanced reasoning framework is necessary to handle such cases, and we leave it as future work.
Commonsense Knowledge Required The given passage might not provide sufficient information. For example, in the passage "He was preparing the paperwork for the move, following the course of an absolutely standard transfer. Sadly he killed himself at home in the meantime," although it states that "preparing the paperwork" and "he killed himself" happened "in the meantime", commonsense knowledge indicates that one cannot kill himself and prepare the paperwork at the same time. So we can infer that "preparing" happened before "killed". Incorporating external knowledge is a potential solution for such cases.
Ambiguous Labeling Since the concept of event is not well-defined, it might lead to ambiguous labeling. Considering a passage contains a span "decision is made", some annotators might label decision as a candidate event, while others does not. This causes inconsistent labeling, and thus makes it difficult to learn a good predictor.
Temporal reading comprehension plays a critical role in natural language understanding. In this paper, we propose a precise question understanding method to tackle the TRC problem. Specifically, we encode temporal ordering questions into repre
sentations of referred events and concerned temporal relations, based on which candidate answers are evaluated in terms of their temporal relations to the referred events. In addition, a contrastive loss is employed to empower the model to capture essential differences among temporal relations. Experimental results based on four pre-trained models verify the effectiveness of our proposed approach. In the future, we will investigate general approaches to handle more diverse temporal relation understanding problems and improve the passage understanding capability for temporal reading comprehension.
Taylor Cassidy, Bill McDowell, Nathanael Chambers, and Steven Bethard. 2014. An annotation framework for dense event ordering. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, ACL 2014, June 22-27, 2014, Baltimore, MD, USA, Volume 2: Short Papers, pages 501-506. The Association for Computer Linguistics.
Kunlong Chen, Weidi Xu, Xingyi Cheng, Zou Xiaochuan, Yuyu Zhang, Le Song, Taifeng Wang, Yuan Qi, and Wei Chu. 2020. Question directed graph attention network for numerical reasoning over text. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 6759-6768. Association for Computational Linguistics.
Fei Cheng, Masayuki Asahara, Ichiro Kobayashi, and Sadao Kurohashi. 2020. Dynamically updating event representations for temporal relation classification with multi-category learning. In *Findings of the Association for Computational Linguistics: EMNLP* 2020, Online Event, 16-20 November 2020, volume EMNLP 2020 of *Findings of ACL*, pages 1352-1357. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 4171-4186. Association for Computational Linguistics.
Bhuwan Dhingra, Hanxiao Liu, Zhilin Yang, William W. Cohen, and Ruslan Salakhutdinov. 2017. Gated-attention readers for text comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 - August 4, Volume 1: Long Papers, pages 1832-1846. Association for Computational Linguistics.
Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. 2019. Unified language model pre-training for natural language understanding and generation. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 13042-13054.
Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Proceedings of the 2019 Conference of the North American Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 2368-2378. Association for Computational Linguistics.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. Simcse: Simple contrastive learning of sentence embeddings. CoRR, abs/2104.08821.
Rudolf Kadlec, Martin Schmid, Ondrej Bajgar, and Jan Kleindienst. 2016. Text understanding with the attention sum reader network. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers. The Association for Computer Linguistics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692.
Qiang Ning, Zhili Feng, Hao Wu, and Dan Roth. 2019. Joint reasoning for temporal and causal relations. CoRR, abs/1906.04941.
Qiang Ning, Hao Wu, Rujun Han, Nanyun Peng, Matt Gardner, and Dan Roth. 2020. Torque: A reading comprehension dataset of temporal ordering questions. arXiv preprint arXiv:2005.00242.
Qiang Ning, Hao Wu, and Dan Roth. 2018. A multi-axis annotation scheme for event temporal relations. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers, pages 1318-1328. Association for Computational Linguistics.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21:140:1-140:67.
Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don't know: Unanswerable questions for squad. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics,
ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 2: Short Papers, pages 784-789. Association for Computational Linguistics.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,  $000+$  questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016, pages 2383-2392. The Association for Computational Linguistics.
Naushad UzZaman, Hector Llorens, Leon Derczynski, James F. Allen, Marc Verhagen, and James Pustejovsky. 2013. Semeval-2013 task 1: Tempeval-3: Evaluating time expressions, events, and temporal relations. In Proceedings of the 7th International Workshop on Semantic Evaluation, SemEval@NAACL-HLT 2013, Atlanta, Georgia, USA, June 14-15, 2013, pages 1-9. The Association for Computer Linguistics.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 5754-5764.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. Hotpotq: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018, pages 2369-2380. Association for Computational Linguistics.
Ben Zhou, Kyle Richardson, Qiang Ning, Tushar Khot, Ashish Sabharwal, and Dan Roth. 2021. Temporal reasoning on implicit events from distant supervision. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021, pages 1361-1371. Association for Computational Linguistics.
A Supplement Information for Experiments
A.1 Dictionary for Temporal Relations
['before', 'after', 'while', 'not', 'future', 'might', 'happen', 'will', 'may', 'have', 'begin', 'but', 'fin-
ish', 'don't', 'continue', 'do', 'start', 'eventually', 'during', 'likely', 'needs', 'occur', 'take', 'place', 'lead', 'when', 'prior', 'same', 'time', 'end', 'ongoing', 'now', 'past', 'since', 'already', 'expect', 'go', 'fail', 'around', 'once', 'be']