SlowGuess/ACL_2020_OCR huggingface dataset:
The label imbalance in the training data largely hinders our goal, as we should not assume a prior distribution as expressed in natural language. For example, "seconds" appears around ten times less than "years" in the data we collected for duration, leading to a biased model. We use weight adjustment to fix this. Specifically, we apply weight adjustment to the total loss with a weight factor calculated as the observed label's count relative to the number of all instances.
4.1 Variations and Settings
We experiment with several variants of the proposed system to study the effect of each change.
Input Size. A model with three input sentences (including the event sentence's left/right neighbors) are labeled with MS. Non MS models use only one sentence in which the event occurs.
All Event Masking. A model with  $p_{\text{event}} = 0.6$  is labeled as AM, and  $p_{\text{event}} = 0.15$  otherwise.
Final Model. Our final model includes all auxiliary dimensions (AUX) (mentioned in §3.2), uses soft cross-entropy loss (SL) and applies weight adjustment (ADJ) (mentioned in §3.6). We study each changes' effect by ablating them individually.
To deal with the skew present in the training data (§3), we down-sample to ensure roughly the same occurrences of each dimension (except for frequency because of its low quantity). As a result, 4.3 million sentences were used in pre-training (down-sampled from 25 million mined sentences). We employ a learning rate of 2e-5 with 3 epochs and set  $p_{mask} = 0.6$  and  $p_{dim} = 0.1$ . Other parameters are the same as those of the BERT base model. We use epoch 2's model for extrinsic evaluations to favor generalization, and epoch 3's model for intrinsic evaluations as it achieves the best performance across tasks.
4.2 Intrinsic Evaluation
We evaluate our method on the temporal value recovery task, where the inputs are a sentence representing the event, an index to the event's verb, and a target dimension. The goal is to recover the temporal value of the given event in the given dimension.
Datasets. To ensure a fair comparison, we sample instances from a new corpus RealNews (Zellers et al., 2019) that have no document overlap with
7584
our pre-training data and, at the same time making the data not strictly in-domain. We apply the same pattern extraction process mentioned in §3.3 on the new data and collect instances that are uniformly distributed across dimensions and values. In addition, we ask annotators on Mechanical Turk to filter out the events that cannot be recovered by common sense. For example, "I brush my teeth [Month]" will be discarded because all candidate answers are approximately uniformly distributed so that one cannot identify a subgroup of labels to be more likely.
Specifically, we ask one annotator to select from 4 choices regarding each (event, temporal value) tuple. The choices are 1) the event is unclear and abstract; 2) the event has a uniform distribution across all labels within the dimension; 3) the given label is one of the top  $25\%$  choices among all other labels within the dimension and 4) the given label is not very likely. We keep the instances for which the annotator selects option 3), verifying that the label is a very likely choice for the given dimension. For the RealNews corpus, we annotate 1,774 events that are roughly uniformly distributed across dimensions and labels, among which 300 events are preserved.
We also apply the same process to UDST dataset. We find the majority of the original annotation to be unsuitable, as there are many annotations to events that are seemingly undecidable by common sense. We first apply an initial filtering by using only events of which the anchor word is a verb and require all existing annotations from (Vashishtha et al., 2019) of the same instance to have an average distance less than two units. We then use our method to annotate 1,047 events, and eventually, 142 instances are left.
Systems. In both datasets, we compare our proposed system with BERT. To use BERT's predictions on temporal values without supervision, we artificially add prepositions querying the target dimension as well as a masked token right after the verb. For example, "I ran to the campus" will be transformed as "I ran for 1 [MASK] to the campus". The specific prepositions added are "for 1" (duration), "every" (frequency), "in the" (time of the day), "on" (week), "in" (month), and "in" (season). We then rank the temporal keywords (singular) in the given dimension according to the masked token's predictions. For our model, we follow the sequence formulation described above, recover and rank the masked [Val] token.
In addition, we also compare with a baseline system called  $BERT +$  naive finetune, which is BERT fine-tuned on the same pre-training data we used for our proposed models, with a higher probability of masking a temporally related keyword (i.e., all values we used in all dimensions). Unlike our model, we only use soft cross-entropy loss and do not distinguish the dimensions each keyword is expressing.
Metrics. Following Vashishtha et al. (2019), we employ a metric "distance" that measures the rank difference between a system's top prediction and the gold label with respect to an ordered label set. For duration and frequency where values are in a one-directional order, we use the absolute difference of the label ranks. For other dimensions where the labels are in circular relationships, we use the minimal distance between two labels in both directions, so that "January" will have a distance 1 with "December." This is similar to an MAE metric, and we report the averaged number across instances.
Figure 5: Representations of events (whose durations were labeled as seconds, weeks, or centuries) obtained from the original BERT base model.
Figure 6: Representations of the same set of events as in Fig. 5 obtained from the proposed method.
7585
Systems	RealNews	UDS-T
Typical Time	Duration
Duration	Freq	Day	Week	Month	Season
BERT	1.33	1.68	1.75	1.53	3.78	0.87	1.77
BERT + naive finetune	1.21	1.45	1.47	1.28	3.28	1.08	2.06
TACOLM (ours)	0.75	1.17	1.72	1.19	3.42	0.63	1.49
TACOLM (ours), normalized	8%	13%	22%	17%	29%	16%	17%
TACOLM -ADJ	0.84	1.20	1.82	1.08	2.47	0.74	1.68
TACOLM -SL	0.77	1.30	1.88	1.06	2.50	0.74	1.50
TACOLM -AUX	0.77	1.28	1.61	1.31	3.25	0.78	1.51
TACOLM -MS	0.84	1.12	1.82	1.5	3.17	0.61	1.69
TACOLM -AM	0.68	1.20	1.86	1.31	3.11	0.70	1.58
The results on the filtered RealNews dataset and filtered UDST dataset are shown in Table 1. We see that our proposed final model is mostly better than other variants, and achieves  $19\%$  improvement over BERT on average on the normalized scale.
We plot the embedding space of events with duration of "seconds" "weeks" or "centuries" in Fig 5 and Fig 6. We take the verb's contextual representation, apply PCA to reduce the dimension from 768 to 50, and then t-SNE to reduce it further to 2. Comparing the two plots, we see that the clusters formed by BERT embeddings have a wider distribution over the space, and the clusters have more points in overlap, even though the three sets of events have drastically different duration values. Our proposed model's embedding is able to better cluster the events based on this temporal feature, which is expected.
4.3 TimeBank Evaluation
Beyond unsupervised intrinsic experiments, we also evaluate the capability of the event temporal representation as a product of our model. That is, we finetune both BERT baseline and our model with the same process to compare the internal representations of the transformers. We use TimeML (Saurei et al., 2005; Pan et al., 2006), a dataset with event duration annotated as lower and upper bounds. The task is to decide whether a given event has a duration longer or shorter than a day. This is a suitable task to evaluate the embeddings because deciding longer/shorter than a day requires reasoning with more than one label, and would also benefit from auxiliary dimensions like duration upper-bound.
The dataset contains 2,251 events, and we split the events based on sentences into 1,248/1,003
train/test. We formulate the training as a sequence classification task by taking the entire sentence and adding a special marker to the left of the verb indicating its position. The marker is unseen to both BERT and our model. We use the transformer output of the first token and feed it to an MLP for classification. We use a learning rate of 5e-5 and train for 3 epochs, and we repeat every reported number with 3 different random initialization and take the average.
Table 1: Performance on intrinsic evaluations. The "normalized" row is the ratio of the distance to the gold label over the total number of labels in each dimension. Smaller is better.
System F1	Accuracy	<Day F1	≥Day F1
BERT	73.7	63.7	79.0
TACOLM	81.7	74.8	85.6
Table 2: Performance on TimeBank Classification
Table 2 shows the results of the TimeBank experiment. We see around  $7 - 11\%$  improvement over BERT on this task. Comparing with the state-of-the-art (Vempala et al., 2018) with a different training/testing split, our model is within  $1.5\%$  of the best results but uses  $25\%$  less training data.
4.4 Subevent Relation Extraction
We apply our event representations to the task of event sub-super relation extraction. This is a proper evaluation because the task naturally benefits from temporal commonsense knowledge. Intuitively, short duration or high frequency indicates the event being at a lower hierarchy and vice versa. We test if the temporal focused event representations will improve.
We use HiEVE (Glavaš et al., 2014), a dataset with annotations of four event relationships: no relation (NoRel), coreference (Coref), Child-Parent (C-P) and Parent-Child (P-C). There is no official split for this dataset, so we randomly 80/20 split
7586
the data at the document level and down-sample negative NoRel instances with a probability of 0.4.
Similarly, we formulate the problem as a sequence classification task, where two events are put into one sequence separated by “[SEP],” and verbs are marked by adding a marker token to their left. We use the representations of the first token and feed it to an MLP for classification. We train each model with a 5e-5 learning rate and 3 epochs. Each reported number is an average from 3 runs under different random initialization. During inference time, the probability scores for non-negative relations are averaged from the same event pair's sequences in both orders.
Table 3 shows the results of the HiEVE experiment. We see that TACOLM improves by  $4\%$  and  $8\%$  on the coreference task and the parent-child tasks over BERT, respectively.
Systems F1	NoRel	Coref	C-P	P-C
BERT	90.5	47.9	40.7	40.6
TACOLM	91.3	51.5	49.4	48.5
4.5 Temporal Question Answering
We also evaluate on MCTACO (Zhou et al., 2019), a question answering dataset that requires comprehensive understandings of temporal common sense and reasoning. We compare the exact-match score across the 5 dimensions defined in MCTACO, although this work only focuses on 3 of them. We use the original baseline system and interchange transformer weights to compare between BERT and ours. However, because our model replaces temporal expressions with special tokens, it is at disadvantage to be directly evaluated on the original dataset with temporal expressions in natural language. To fix this, we run the same extraction system in §3.3 with modifications to identify the dimension a question is asking, and augment candidate answers with our special tokens representing the temporal values (if any) mentioned. This introduces rule-based dimension identification as well as coarse unit normalization to the systems, so we train/evaluated BERT baseline with the same modified data as well. Each number is an average of 5 runs with different random initializations.
Results on MCTACO are shown in Table 4. As expected, we find that our model achieves better
Table 3: Performance on HiEVE. The numbers are in percentages. Higher is better.
System	Duration	Ordering	Stationarity	Frequency	Typical Time
BERT	33.4	36.5	57.6	43.3	39.5
TACOLM	34.6	35.1	57.9	45.1	40.9
Table 4: Performance on MCTACO. Numbers are percentages and indicate exact match (EM) metric. Higher is better.
performance on the three dimensions that are focused in this work (i.e., duration, frequency, and typical time) as well as stationarity. However, the improvements are not very substantial, indicating the difficulty of this task and motivates future works. The model also does slightly worse on ordering, which is worth investigating in future works.
Temporal common sense (TCS) is an important yet challenging research topic. Despite the existence of several prior work on event duration, this is the first attempt to jointly model three key dimensions of TCS—duration, frequency, and typical time—from cheap supervision signals mined from unannotated free text. The proposed sequence modeling framework improves over BERT in terms of handling reporting bias, taking into account the ordinal relations and exploiting interactions among multiple dimensions of time. The success of this model is confirmed by intrinsic evaluations on RealNews and UDS-T (where we see a  $19\%$  improvement), as well as extrinsic evaluations on TimeBank, HiEVE and MCTACO. The proposed method may be an important module for future applications related to time.
This research is based upon work supported in part by the office of the Director of National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via IARPA Contract No. 2019-19051600006 under the BETTER Program and by Contract FA8750-19-2-1004 with the US Defense Advanced Research Projects Agency (DARPA). The views expressed are those of the authors and do not reflect the official policy or position of the Department of Defense or the U.S. Government. This research is also supported by a grant from the Allen Institute for Artificial Intelligence (allenai.org).
7587
Gabor Angeli, Christopher D Manning, and Daniel Jurafsky. 2012. Parsing time: Learning to interpret time expressions. In *NAACL-HLT*, pages 446-455.
Hessam Bagherinezhad, Hannaneh Hajishirzi, Yejin Choi, and Ali Farhadi. 2016. Are elephants bigger than butterflies? reasoning about sizes of objects. In AAAI.
Lisa Bauer, Yicheng Wang, and Mohit Bansal. 2018. Commonsense for generative multi-hop question answering tasks. In EMNLP, pages 4220-4230.
Steven Bethard, Guergana Savova, Wei-Te Chen, Leon Derczynski, James Pustejovsky, and Marc Verhagen. 2016. SemEval-2016 Task 12: Clinical TempEval. In SemEval, pages 1052-1062.
Nathanael Chambers, Taylor Cassidy, Bill McDowell, and Steven Bethard. 2014. Dense event ordering with a multi-pass architecture. TACL, 2:273-284.
Timothy Chklovski and Patrick Pantel. 2004. VerbOcean: Mining the Web for Fine-Grained Semantic Verb Relations. In EMNLP, pages 33-40.
Anne Cocos, Veronica Wharton, Ellie Pavlick, Marianna Apidianaki, and Chris Callison-Burch. 2018. Learning scalar adjective intensity from paraphrases. In EMNLP, pages 1752-1762.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In *NAACL*.
Yanai Elazar, Abhijit Mahabal, Deepak Ramachandran, Tania Bedrax-Weiss, and Dan Roth. 2019. How large are lions? inducing distributions over quantitative attributes. In ACL.
Maxwell Forbes and Yejin Choi. 2017. Verb physics: Relative physical knowledge of actions and objects. In ACL, volume 1, pages 266-276.
Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson F Liu, Matthew Peters, Michael Schmitz, and Luke Zettlemoyer. 2018. Allennlp: A deep semantic natural language processing platform. In NLP-OSS, pages 1-6.
Goran Glavaš, Jan Šnajder, Marie-Francine Moens, and Parisa Kordjamshidi. 2014. HiEve: A corpus for extracting event hierarchies from news stories. In LREC, pages 3678-3683, Reykjavik, Iceland. European Language Resources Association (ELRA).
Jonathan Gordon and Benjamin Van Durme. 2013. Reporting bias and knowledge acquisition. In *AKBC*, pages 25-30. ACM.
David Graff, Junbo Kong, Ke Chen, and Kazuaki Maeda. 2003. English gigaword. Linguistic Data Consortium, Philadelphia, 4(1):34.
Mark Granroth-Wilding and Stephen Christopher Clark. 2016. What happens next? event prediction using a compositional neural network model. In ACL.
Andrey Gusev, Nathanael Chambers, Pranav Khaitan, Divye Khilnani, Steven Bethard, and Dan Jurafsky. 2011. Using query patterns to learn the duration of events. In IWCS, pages 145-154.
Luyao Huang, Chi Sun, Xipeng Qiu, and Xuanjing Huang. 2019. GlossBERT: BERT for word sense disambiguation with gloss knowledge. In EMNLP, pages 3507-3512.
Kenton Lee, Yoav Artzi, Jesse Dodge, and Luke Zettle-moyer. 2014. Context-dependent semantic parsing for time expressions. In ACL (1), pages 1437-1447.
Artuur Leeuwenberg and Marie-Francine Moens. 2017. Structured learning for temporal relation extraction from clinical records. In EACL.
Artuur Leeuwenberg and Marie-Francine Moens. 2018. Temporal information extraction by predicting relative time-lines. EMNLP.
Zhongyang Li, Xiao Ding, and Ting Liu. 2018. Constructing narrative event evolutionary graph for script event prediction. *IJCAI*.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Hector Llorens, Nathanael Chambers, Naushad UzZaman, Nasrin Mostafazadeh, James Allen, and James Pustejovsky. 2015. SemEval-2015 Task 5: QA TEMPEVAL - evaluating temporal information understanding with question answering. In SemEval, pages 792-800.
Qiang Ning, Zhili Feng, and Dan Roth. 2017. A structured learning approach to temporal relation extraction. In EMNLP.
Qiang Ning, Zhili Feng, Hao Wu, and Dan Roth. 2018a. Joint reasoning for temporal and causal relations. In ACL.
Qiang Ning, Hao Wu, Haoruo Peng, and Dan Roth. 2018b. Improving temporal relation extraction with a globally acquired statistical resource. In NAACL, pages 841-851.
Qiang Ning, Ben Zhou, Zhili Feng, Haoruo Peng, and Dan Roth. 2018c. CogCompTime: A tool for understanding time in natural language. In EMNLP.
Feng Pan, Ritu Mulkar, and Jerry R Hobbs. 2006. Extending TimeML with typical durations of events. In *ARTE*, pages 38-45. Association for Computational Linguistics.
7588
Haoruo Peng, Qiang Ning, and Dan Roth. 2019. KnowSemLM: A Knowledge Infused Semantic Language Model. In CoNLL.
Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of NAACL-HLT, pages 2227-2237.
James Pustejovsky, Patrick Hanks, Roser Sauri, Andrew See, Robert Gaizauskas, Andrea Setzer, Dragomir Radev, Beth Sundheim, David Day, Lisa Ferro, et al. 2003. The TIMEBANK corpus. In Corpus linguistics, volume 2003, page 40.
Roser Saurei, Jessica Littman, Bob Knippen, Robert Gaizauskas, Andrea Setzer, and James Pustejovsky. 2005. Timeml annotation guidelines.
Lenhart Schubert. 2002. Can we derive general world knowledge from texts? In HLT, pages 94-97. Morgan Kaufmann Publishers Inc.
Peng Shi and Jimmy Lin. 2019. Simple bert models for relation extraction and semantic role labeling. arXiv preprint arXiv:1904.05255.
Jannik Strötgen and Michael Gertz. 2010. Heidel-Time: High quality rule-based extraction and normalization of temporal expressions. In SemEval, pages 321-324. Association for Computational Linguistics.
Niket Tandon, Bhavana Dalvi, Joel Grus, Wen-tau Yih, Antoine Bosselut, and Peter Clark. 2018. Reasoning about actions and state changes by injecting commonsense knowledge. In EMNLP, pages 57-66.
Naushad UzZaman, Hector Llorens, James Allen, Leon Derczynski, Marc Verhagen, and James Pustejovsky. 2013. SemEval-2013 Task 1: TEMPEVAL-3: Evaluating time expressions, events, and temporal relations. *SEM, 2:1-9.
Benjamin Van Durme. 2009. Extracting implicit knowledge from text. Ph.D. thesis, University of Rochester.
Siddharth Vashishtha, Benjamin Van Durme, and Aaron Steven White. 2019. Fine-grained temporal relation extraction. In ACL, pages 2906-2919.
Alakananda Vempala, Eduardo Blanco, and Alexis Palmer. 2018. Determining event durations: Models and error analysis. In *NAACL*, volume 2, pages 164-168.
Jennifer Williams. 2012. Extracting fine-grained durations for verbs from twitter. In ACL-SRW, pages 49-54. Association for Computational Linguistics.
Yiben Yang, Larry Birnbaum, Ji-Ping Wang, and Doug Downey. 2018. Extracting commonsense properties from embeddings with limited human guidance. In ACL, volume 2, pages 644-649.
Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. 2019. Defending against neural fake news. In NeurIPS, pages 9054-9065. Curran Associates, Inc.
Sheng Zhang, Rachel Rudinger, Kevin Duh, and Benjamin Van Durme. 2017. Ordinal common-sense inference. TACL, 5(1):379-395.
Ben Zhou, Daniel Khashabi, Qiang Ning, and Dan Roth. 2019. "Going on a vacation" takes longer than "Going for a walk": A Study of Temporal Common-sense Understanding. In EMNLP.
7589