SlowGuess/ACL_2020_OCR huggingface dataset:
Ben Zhou, $^{1}$  Qiang Ning, $^{2}$  Daniel Khashabi, $^{2}$  Dan Roth $^{1}$
<sup>1</sup>University of Pennsylvania, <sup>2</sup>Allen Institute for AI
{xyzhou,danroth}@cis.upenn.edu {qiangn,danielk}@allenai.org
Temporal common sense (e.g., duration and frequency of events) is crucial for understanding natural language. However, its acquisition is challenging, partly because such information is often not expressed explicitly in text, and human annotation on such concepts is costly. This work proposes a novel sequence modeling approach that exploits explicit and implicit mentions of temporal common sense, extracted from a large corpus, to build TACOLM, $^{1}$  a temporal common sense language model. Our method is shown to give quality predictions of various dimensions of temporal common sense (on UDST and a newly collected dataset from RealNews). It also produces representations of events for relevant tasks such as duration comparison, parent-child relations, event coreference and temporal QA (on TimeBank, HiEVE and MCTACO) that are better than using the standard BERT. Thus, it will be an important component of temporal NLP.
Time is crucial when describing the evolving world. It is thus important to understand time as expressed in natural language text. Indeed, many natural language understanding (NLU) applications, including information retrieval, summarization, causal inference, and QA (UzZaman et al., 2013; Chambers et al., 2014; Llorens et al., 2015; Bethard et al., 2016; Leeuwenberg and Moens, 2017; Ning et al., 2018b), rely on understanding time.
However, understanding time in natural language text heavily relies on common sense inference. Such inference is challenging since commonsense information is rarely made explicit in text (e.g., how long does it take to open a door?) Even when such information is mentioned, it is often
Figure 1: Our model's predicted distributions of event duration and frequency. The model is able to attend to contextual information and thus produce reasonable estimates.
affected by another type of reporting bias: people rarely say the obvious, in order to communicate more efficiently, but sometimes highlight rarities (Schubert, 2002; Van Durme, 2009; Gordon and Van Durme, 2013; Zhang et al., 2017; Bauer et al., 2018; Tandon et al., 2018).
This is an even more pronounced phenomenon when it comes to temporal common sense (TCS) (Zhou et al., 2019). In Example 1, human readers know that a typical vacation is likely to last at least a few days, and they would choose "will not" to fill in the blank for the first sentence; instead, with a slight change of context "vacation"  $\rightarrow$  "walk outside," people typically prefer "will" for the second one. Similarly, any system which correctly answers this example for the right reason would need to incorporate TCS in its reasoning.
Example 1: choosing from "will" or "will not"
Dr. Porter is now (e.l: taking) a vacation and be able to see you soon.
Dr. Porter is now (e2:taking) a walk outside and be able to see you soon.
Acquiring the multiple dimensions of TCS (e.g., duration and frequency) is challenging. As shown
1https://cogcomp.seas.upenn.edu/page/publication_view/904
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7579-7589
in Example 1, the duration of "taking a vacation" and "taking a walk" are not expressed explicitly, so that systems are required to read between the lines to support the inference. A pre-trained language model may not handle this issue well, as it cannot identify the TCS dimensions in temporal mentions and effectively learn from them. As a result, it cannot generalize well to similar events without explicit temporal mentions. To handle this problem, we design syntactic rules that can collect a vast amount of explicit mentions of TCS from unannotated corpus such as Gigaword (Graff et al., 2003) (§3.3). We use this data to pre-train our model so that it distinguishes different dimensions.
A second challenge occurs when the text is highlighting rare and special cases. As a result, temporal mentions in natural text may follow a distorted distribution in which certain kinds of "common" events are under-represented. For instance, we may rarely see mentions of "I opened the door in 3 seconds," but we may see "it took me an hour to open this door" in text. To overcome this challenge, we exploit the joint relationship among temporal dimensions. Although we rarely observe the true duration of "opening the door" in free-form text, we may see phrases like "I opened my door during the fire alarm," providing an upper-bound to the duration of the event (i.e., "opening the door" does not take longer than the alarm.) We believe that we are the first to exploit such phenomena among temporal dimensions.
This paper studies several important dimensions of TCS inference: duration (how long an event takes), frequency (how often an event occurs) and typical time (when an event typically happens). As a highlight, Fig. 1 shows the distributions (over time units) we predict for the duration and frequency of three events. We can see that "taking a vacation" lasts from days to months while "taking a walk" lasts from minutes to hours. As shown, our model is able to produce different and sensible distributions for the "take" event, depending on the context in which "take" occurs.
Our work builds upon pre-trained contextual language models (Peters et al., 2018; Devlin et al., 2019; Liu et al., 2019). However, a standard language modeling objective does not lead to a model that handles the two challenges mentioned above; in addition, other systematic issues limit its ability to handle TCS. In particular, language models do
not directly utilize the ordinal relationships among temporal units. For example, “hours” is longer than “minutes,” and “minutes” are longer than “seconds.”<sup>3</sup> Fig. 2 shows that BERT does not produce a meaningful duration distribution for a set of events with a gold duration of “day” (extracted in §3.3). Our proposed system, on the other hand, is able to utilize the ordinal relationships and produce unimodal distributions around the correct labels in both Fig. 1 and Fig. 2.
Figure 2: The predictive distribution of two models (ours and vanilla BERT) for a set of events labeled with "days" as their duration. Experiments show our model is about  $40\%$  better on duration predictions. (RealNews corpus; details in  $\S 4.2$ ).
Contributions. This work proposes an augmented pre-training for language models to improve their understanding of several important temporal phenomena. We address two kinds of reporting biases by effectively acquiring weak supervision from free-form text and utilizing it to learn multiple temporal dimensions jointly. Our model incorporates other desirable properties of time in its objective (ordinal relations between temporal phrases, the circularity of certain dimensions, etc.) to improve temporal modeling. Our experiments show  $19\%$  relative improvement over BERT in intrinsic evaluations, and  $5 - 10\%$  improvements in most extrinsic evaluations done on three time-related datasets. Furthermore, the ablation study shows the value of each proposed component of our construction. Overall, this is the first work to incorporate a wide range of temporal phenomena within a contextual language model.
The rest of this paper is organized as follows. We distinguish our work with the prior work in §2. The core of our construction, including extraction of cheap supervision from raw data and augmenting a language model objective function with temporal signals, is in §3. We conclude by showing intrinsic and extrinsic experiments in §4.
$^{2}$ E.g., typical time in a day (the morning), typical day of a week (on Sunday), and typical time of a year (summer).
3The relationship can be more complex. E.g., "hours" is closer to "minutes" than "centuries" is; days of a week forms a circle: "Mon." is followed by "Tue." and preceded by "Sun."
Common sense has been a popular topic in recent years, and existing NLP works have mainly investigated the acquisition and evaluation of common sense reasoning in the physical world. These works include but are not limited to size, weight, and strength (Bagherinezhad et al., 2016; Forbes and Choi, 2017; Elazar et al., 2019), roundness and deliciousness (Yang et al., 2018), and intensity (Cocos et al., 2018). A handful of these works uses cheap supervision. For example, Elazar et al. (2019) recently proposed a general framework that discovers distributions of quantitative attributes (e.g., length, mass, speed, and duration) from explicit mentions (or co-occurrences) of these attributes in a large corpus. However, Elazar et al. (2019) restrict events to be verb tokens, while we handle verb phrases containing more detailed information (e.g., "taking a vacation" is very different from "taking a break," although they share the same verb "take"). Besides, there has been no report on the effectiveness of this method on temporal attributes.
On the other hand, time has long been an important research area in NLP. Prior works have focused on the extraction and normalization of temporal expressions (Strötgen and Gertz, 2010; Angeli et al., 2012; Lee et al., 2014; Vashishtha et al., 2019), temporal relation extraction (Ning et al., 2017, 2018c; Vashishtha et al., 2019), and timeline construction (Leeuwenberg and Moens, 2018). Recently, MCTACO (Zhou et al., 2019) summarizes five types of TCS and the three temporal dimensions studied here are all in their proposal. MCTACO shows that modern NLU techniques are still a long way behind humans on TCS understanding, suggesting that further research on this topic is needed.
There have been works on temporal common sense, such as event duration (Pan et al., 2006; Gusev et al., 2011; Williams, 2012; Vempala et al., 2018; Vashishtha et al., 2019), typical temporal ordering (Chklovski and Pantel, 2004; Ning et al., 2018a,b), and script learning (i.e., what happens next after certain events) (Granroth-Wilding and Clark, 2016; Li et al., 2018; Peng et al., 2019). Those on duration are highly relevant to this work. (Pan et al., 2006) annotates a subset of documents from TimeBank (Pustejovsky et al., 2003) with
"less-than-one-day" and "more-than-one-day" annotations and provide the first baseline system for this dataset. Vempala et al. (2018) significantly improve earlier work by using additional aspectual features for this task. Vashishtha et al. (2019) annotate the UDS-T dataset with event duration annotations and propose a joint method that extracts both temporal relations and event durations. Our approach has two notable differences from this line of work. First, we work on duration, frequency, and typical time—jointly on three dimensions of TCS, while the works above only focused on duration. Second, we focus more on obtaining cheap supervision signals from unlabeled data, while these other works all have access to human annotations. With respect to harnessing cheap supervision, Williams (2012); Gusev et al. (2011) propose to mine web data using a collection of hand-designed query patterns. In contrast to our approach, they are based on counting instead of machine learning and cannot handle the contextualization of events.
3 Temporally Focused Joint Learning with Minimal Supervision
In this section, we elaborate our approach to designing and pre-training TACOLM, a time-aware language model.
3.1 Scope
In this work, we focus on three major temporal dimensions of events, namely Duration, Frequency and Typical Time. Here, Typical Time means the typical occurring time of events during a day, day of a week, and month or season of a year. We follow the same definition to each of the dimensions (also called properties) in Zhou et al. (2019).
3.2 Joint Learning and Auxiliary Dimensions
As mentioned earlier, commonsense information extraction comes with the challenge of reporting biases. For example, people may not report the duration of "opening the door," or the frequency of "going to work." However, it is often possible to get supportive signals from other dimensions, as people mention "going to work" associated mostly with "a day" in a week, hence we may know the frequency of such an event.
We argue that many temporal dimensions are interrelated and a joint learning scheme would suit this task. Beyond duration, frequency and typical time, we also introduce auxiliary dimensions that are not meant to be used by themselves but will
<sup>4</sup>They additionally propose typical order of events and stationarity (whether a state holds for a very long time or indefinitely).
7581
Figure 3: Examples of the extraction process for each temporal dimensions. The temporal arguments are marked orange and the result of the extraction are tuples of the form (event, value, dimension).
help the prediction of other dimensions. The auxiliary dimensions we define here are event Duration Upper-bound and event Relative Hierarchy. The former represents values that are upper-bounds to an event's duration but not necessarily the exact duration. The latter consists of two sub-relations, namely temporal ordering and duration inclusion of event-pairs.
3.3 Cheap Supervision from Patterns
We collect a few pattern-based extraction rules based on SRL parses for each temporal dimension (including the auxiliary dimensions). We design the rules to have high precision, while not compromising too much on recall. We overcome the potential sparsity issue (and the resulting low recall problem) by extracting from a massive amount of data. Fig. 3 provides some examples of the input/output for each dimension, as we describe the specific extraction process below.
We first process the entire Gigaword (Graff et al., 2003) corpus and use AllenNLP's SRL model (Gardner et al., 2018; Shi and Lin, 2019) to annotate all sentences. We extract the ones that contain at least one temporal argument (i.e., the arg-temp constituent of SRL annotations) and use textual patterns to categorize each sentence into a corresponding dimension with respect to an associated verb. These patterns are inspired by earlier works and are extensively improved with iterative manual error analysis. The rest of this section is devoted to explaining the key design ideas used for these patterns.
Duration. We check if the temporal argument starts with "for," extract the numerical value and
the temporal unit word, and normalize them into the nearest unit among the nine units in our scope: ("second," "minute," "hour," "day," "week," "month," "year," "decade," "century.") We ignore particular phrases such as "for a second chance" where the semantic of "second" is not temporal related. We found that "for" is the only high-precision preposition that indicates exact values of duration.
Frequency. Such temporal arguments are usually composed of a duration phrase and a numerical head (e.g., "four times per") indicating the frequency within the duration (e.g., "week"). Thus, we check for multiple keywords that indicate the start of a frequency expression, including "every," "per," "once," ... "times." If so, we extract the duration value as well as the numerical head's value. We ignore any temporal phrases that contain "when" since they often convey semantics that does not fit any of our temporal categories; e.g., "when everyday life..." is not describing the frequency of the corresponding verb. We represent the frequency with duration d, with a definition of occurring once every d elapses. For example, the frequency of "four times per week" is represented as "1.75 days." Similarly, we normalize them into the nearest unit among the nine duration units described above, and "1.75 days" is extracted as "days."
Typical Time. We pre-define a list of typical time keywords, including the time of day (e.g., "morning" etc.), time of week (e.g., "Monday" etc.), month (e.g., "January" etc.) and season (e.g., "winter" etc.) We check if any of the typical time keywords appear in the temporal argument and verify if the temporal argument is, in fact, describing the time of occurrence. This is done by filtering out the temporal arguments that contain a set of invalid prepositions, including "until," "since," "following," since such keywords often do not indicate the actual time of occurrence.
Duration Upper-bound. Many temporal arguments describe the duration upper-bound instead of the exact duration value. For example, as described in (Gusev et al., 2011), "did [activity] yesterday" indicates something that happened within a day. We extend the set of patterns to include "in [temporal expression]" or keywords such as "next" (e.g., "the next day"), "last" (e.g., "last week"), "previous" (e.g., "previous month"), or "recent" (e.g., "recent years"). We normalize the values into the same label set of the nine unit words as the duration dimension.
7582
Event Relative Hierarchy. A system can learn about an event with comparisons to other ones, as we show in §1. To acquire hierarchical relationships between events, we check whether the SRL temporal argument starts with a keyword that indicates a relation between the main event and another event phrase. We consider five such keywords, namely "before," "after," "during," "while" and "when." We use these keywords to label the relative relationship between the two events. Here, we assume that "during" and "while" are the same, which indicates that the main event is not longer than the one in the argument. Note that certain keywords might have meanings that do not suggest temporal relationships (e.g., "while" has a different sense similar to "whereas.") We rely on SRL annotations to identify the appropriate sense of the keywords. We use the temporal keyword as labels, but keep the entire event phrase in the SRL temporal argument for later use in §3.5.
Resulting data. We collect 25 million instances that are successfully parsed into one of our temporal dimensions from the entire Gigaword corpus (Graff et al., 2003). Each instance is in the form of (event, value, dimension) tuples (Fig. 3), with a dimension distribution shown in Fig. 4. For all events, we remove the related temporal argument so that it does not contain direct information about the dimension or the value. For example, as shown in Fig. 3, "for 2 hours" is removed, and only "Jack rested before the speech" is kept so that the target duration does not present in the event. Note that value is also called and used as "label" in later contexts related to classification tasks.
Figure 4: The distributions of different temporal dimensions in the collected data.
3.4 Soft Cross-Entropy Objective for Ordinal Classification
The temporal values in one dimension are naturally related to each other via a certain ordering and appropriate distance measures. To account for and utilize this external knowledge, we use a soft cross-entropy to encourage predictions that are aligned
with the external knowledge.
Consider  $\mathbf{x}$  as a system's output logits across labels, and we express our soft loss function as follows:
$$
\ell = - \sum_ {i \in D} \mathbf {y} _ {i} ^ {\top} \log \left(\operatorname {s o f t m a x} \left(\mathbf {x} _ {i}\right)\right), \tag {1}
$$
where  $D$  is the instances in the training data and  $\mathbf{y}$  represent the degree to which the target labels align with the external knowledge. Thus,  $\mathbf{y}$  is a probability vector, i.e., has non-zero values and sum to 1.0.
Now we describe how we construct  $\mathbf{y}$  to apply the aforementioned external knowledge. Duration, Frequency, and Upper-bound take the same set of labels of duration units. We first define a function  $\operatorname{logsec}(.)$  which takes a unit and normalizes it to its logarithmic value in "seconds" (e.g., "minute"  $\rightarrow$  60  $\rightarrow$  4.1). For each instance in these dimensions, with an observed gold label  $g$ , we assume a normal distribution with a mean value of  $\mu = \operatorname{logsec}(g)$  and a fixed standard deviation of  $\sigma = 4$ . Then, we construct  $\mathbf{y}$  so that,
$$
\mathbf {y} [ i ] = \frac {1}{\sigma \sqrt {2 \pi}} e ^ {- (\log \sec (l) - \mu) ^ {2} / 2 \sigma^ {2}} \tag {2}
$$
where  $l$  is the  $i^{th}$  label. We apply softmax at the end to ensure  $\mathbf{y}$  sums to 1.
For typical time, the labels are placed with approximately equal distances in a circular fashion. For example, "Monday" is before "Tuesday" and after "Sunday." We assume adjacent units have a distance of 1, and we generate  $y$  based on a Gaussian distribution with a standard deviation of 0.5. In other words, we assume the two immediate neighbors of a gold label are reasonably possible.
For hierarchy, we construct  $\mathbf{y}$  as a one-hot vector where only the gold label has a value of 1, and the rest are zeroes.
3.5 Sequential Language Modeling
Our goal is to build a model that is able to predict temporal labels (values) given events and dimensions. Instead of building a classification layer on top of a pre-trained model, we follow previous work (Huang et al., 2019) and place the label into the input sequence. We mask the label in the sequence and use the masked token prediction objective as the classification objective. To produce more general representations, we also keep the temporal label and mask the event tokens instead at a certain probability, so that we are
7583
able to maximize both  $\mathbb{P}(\text{Tmp-Label|Event})$  and  $\mathbb{P}(\text{Event|Tmp-Label})$  in the same learning process, where  $\text{Tmp-Label}$  refers to the temporal label associated with the event.
Specifically, we use the reserved "unused" tokens in BERT-base model lexicon to construct a 1-to-1 mapping from every value in every dimension to the new vocabulary. We choose not to use the existing representations for temporal terms that are already included in BERT's "in-use" lexicon, such as "minutes" or "weeks," because these keywords have different temporal semantics in different dimensions. Instead, we assign unique and separate lexicon entries to different values in different dimensions, even though the values may share the same surfaces. Consider each (event, value, dimension) tuple, we map value and dimension to their new vocabularies [Val] and [Dim], and we use  $\left[\mathrm{W}_{1},\mathrm{W}_{2},\dots ,\mathrm{W}_{n}\right]$  to represent the tokens in the sentence, and WVerb the event verb anchor from SRL. We now form a sequence  $\left[\mathrm{W}_1,\mathrm{W}_2,\ldots \left[\mathrm{Vrb}\right],\mathrm{W_{verb}},\ldots \mathrm{W}_n,\left[\mathrm{SEP}\right],\right.$  [Vrb], [Dim], [Val], [Arg-Tmp-Event]], where [Vrb] is a marker token that is the same across all instances. [Arg-Tmp-Event] is the event phrase in the SRL temporal argument, as described in hierarchy. [Arg-Tmp-Event] is empty for all dimensions other than hierarchy.
We mask [Val] with probability  $p_{mask}$  and [Dim] with probability  $p_{dim}$ . We individually mask each event tokens with probability  $p_{event}$  when we do not mask [Val] nor [Dim]. Soft cross-entropy is used when predicting [Val], and a regular Cross-entropy is used for other tokens. We use the pre-trained token-recovery layer, and follow BERT's setting to randomly keep a token's surface or change it to noise during recovery.
In the experiments, we explore a set of configurations of the system. We explore the effect of having only one sentence or the two additional neighboring sentences as input contexts. We also experiment with all-event-masking, where we mask tokens in the event with a much higher probability. The goal of this masking scheme is to reduce the predictability of event tokens based on other event tokens to alleviate prior biases and focus more on the temporal argument. For example, BERT predicts "coffee" for the [MASK] in "I had a cup of [MASK] this evening" because of the strong prior of "cup of." By masking more tokens in the event, the remaining ones will be more conditioned to the temporal cue.
3.6 Label Weight Adjustment