Open Temporal Relation Extraction for Question Answering

1 . Introduction . Despite the significant progress made in question answering ( QA ) over the recent years [ Gupta and Gupta , 2012 , Mishra and Jain , 2016 , Höffner et al. , 2017 ] , questions that involve temporal reasoning between events in text have received relatively little attention . Previous work has shown that many existing QA techniques are usually ill-equipped to tackle the problem of temporal reasoning [ Zhou et al. , 2019 , Ning et al. , 2020 ] . This failure is partly due to the insensitivity of these QA models to nuanced and subtle textual changes in temporal questions that could imply completely different relations . For instance , Figure 1 , “ what happened right after the election ” , “ what failed to happen after the election ” , and “ what happened right before the election ” can have mutually exclusive sets of answer events given the same context . Most previous approaches towards temporal question answering focus on settings where explicit timestamps ( e.g. , “ August 2020 ” ) are available in the question , the context , and/or knowledge bases [ Jin et al. , 2020 , Harabagiu and Bejan , 2005 , Jia et al. , 2018b , a ] . However , explicit timestamps are not available in many texts , where it is still important to understand the temporal relation between events ( e.g. , novels , news reports ) . In these cases , one would need to seek clues from the context in which events are described to understand the relative temporal ordering of events to answer questions ( see Figure 1 for an example ) . Consequently , this task is more taxing for human annotators which limits the amount of annotated data available . On the other hand , the potential set of questions asked is also large thanks to its text-based nature , which is further exacerbated by the tight coupling between questions and the context . Both of these characteristics render this task difficult for powerful onesize-fits-all neural network models since these models are prone to overfitting . In this paper , we propose to reformulate the problem of temporal question answering as one of open temporal relation extraction . Specifically , we decompose each question into an open temporal relation ( OTR ) expressed in natural language ( similar to that of open information extraction [ Etzioni et al. , 2008 ] ) , as well as a question event that is contextdependent . This formulation ( OTR-QA ) has two advantages . Firstly , it allows us to model temporal relations in a context-agnostic manner , which shares supervision signal from different contexts and events to the same underlying open temporal relation . As a result , OTR-QA is much more data-efficient compared to its BERT-based counterpart that does not explicitly consider this decomposition , and generalizes better with the same amount of training data . We demonstrate that OTR-QA significantly improves upon the previous state of the art on the TORQUE [ Ning et al. , 2020 ] dataset . Secondly , this reformulation allows us to explicitly model the differences in temporal relations with a contrastive loss function , which helps capture mutually exclusive relations ( e.g. , Q1 , Q2 , and Q3 in Figure 1 ) and relations whose differences are more nuanced ( e.g. , Q6 and Q7 in Figure 1 , where the answer set of Q7 is a subset of that of Q6 ) . This results in much more coherent answers across questions that are within contrast groups defined in TORQUE , which are usually questions centered around the same question event but have different answers given the same context ( e.g. , the green and blue dashed boxes in Figure 1 is each a contrast group ) . To recap , our main contributions in this paper are : 1 ) we propose to reformulate temporal question answering as a new task , open temporal relation extraction , which is more data-efficient compared to its counterparts that do not factorize the problem ; 2 ) we further show that this formulation allows us to explicitly model contrasting temporal relations with a contrastive loss function , which further improves the coherence of model prediction ; 3 ) our proposed model outperforms the previous state of the art on the TORQUE dataset by a large margin . 2 . Related Works . Time information is crucial for understanding the events and temporal relations among events . Existing studies have explored the time information from different perspectives . Some work made use of the duration and frequency of events for temporal commonsense reasoning [ Vempala et al. , 2018 , Zhou et al. , 2019 , 2020 ] ; some work focused on temporal relation extraction [ Ning et al. , 2019a , b ] , timeline construction [ Leeuwenberg and Moens , 2018 ] and temporal question answering [ Ning et al. , 2020 ] . Recent work DEER [ Han et al. , 2020 ] took the pre-training approach that trained a language model to focus on event temporal relations . A large amount of training samples were created to simulate the QA and information extraction tasks for event temporal understanding . Temporal Relation Types . The attempts on temporal relation extraction are usually making pairwise decisions between each pair of events given a pre-defined temporal relation . Existing approaches for temporal processing often used the interval representation of events proposed in Allen [ 1984 ] which includes 13 relation types in total . When a relation is not clear , a vague or none relation could be also added as another relation type . In CAEVO [ Chambers et al. , 2014 ] , the defined temporal relation types are { before , after , includes , is included , equal , vague } . These approaches all used a predefined set of relation types . However , in the temporal question answering task , the temporal relations included in questions could be diverse that can not be specified in advance . In this paper , we design the model to handle the open temporal relations without defining the pre-defined types . Temporal Question Answering ( TQA ) . Most of TQA work required timestamps in the input . For example ForecastQA [ Jin et al. , 2020 ] formulates the forecasting problem as a multiple-choice question answering task , where both the articles and questions include the timestamps . It introduces a timestamp constraint per question that prohibits the model from accessing articles published after the timestamp . Another task is temporal question answering over knowledge bases ( KB ) [ Jia et al. , 2018b , a ] , which retrieves time information from the KB . Few work has been done to explore the temporal relations without timestamps . The recent released TORQUE [ Ning et al. , 2020 ] is a designed dataset that explores the temporal ordering relations between events described in a passage of text . However , understanding these temporal relations is challenging as they can not be pre-defined in advance , and is also sensitive to the small changes on implicit temporal keywords . To handle these challenges , we propose the OTR-QA model in this paper . What ’ s more , there are few approaches that attempt to train the model with closely related questions , which drawing the ideas from the contrastive learning [ Oord et al. , 2018 ] . Both [ Asai and Hajishirzi , 2020 ] and [ Dua et al. , 2021 ] take advantage of the closely related questions to figure out the difference between the inputs that leads to the expected difference between their answer . Our OTR-QA also captures the temporal difference between the related temporal questions by proposing a contrastive loss function . 3 . Methodology . In this section , we first formulate the open temporal relation extraction problem and then introduce our model that builds on this formulation for temporal question answering . 3.1 Problem Definition . Temporal Order Question Answering . The task of temporal order question answering ( TORQUE ) [ Ning et al. , 2020 ] is a question answering task given a paragraph of text as context . Formally , let C = { s1 , · · · , sn } be a piece of text with n sentences as context , a set of m events , E = { e1 , · · · , em } are defined , which covers salient verbs ( such as “ investigating ” and “ said ” ) and nouns ( such as “ accident ” and “ landslide ” ) . The task of TORQUE aims at answering questions Q that concern with the temporal order of these events where the answer event set Ae ⊂ E ( see Figure 1 for examples ) . Semantically , questions in TORQUE largely take the form of expressing the temporal relation of events in C with respect to a question event eq ∈ E. For instance , the first question in Figure 1 queries for events that “ happened right after ” the event “ election ” , which is one of the highlighted events in the context . Since these questions are written in natural language , they can express nuanced temporal relations such as “ happened right after ” , “ might happen after ” , or “ might have happened before ” , which is difficult to enumerate or define ahead of time . We therefore treat these as open temporal relations ( OTR ) , and propose to answer these questions by extracting these OTRs . Open Temporal Relation Extraction . Given a question Q based on context C with event set E , we begin by decomposing Q into the open temporal relation r it expresses , and the question event eq ∈ E. The task of open temporal relation extraction ( OTRE ) is then to find all candidate events ea ∈ E for which the temporal relation triple ( eq , r , ea ) holds . Once this is done , we have recovered the desired answer set Ae = { ea|ea ∈ E and ( eq , r , ea ) ∈ TemporalRelations ( C ) } .1 This new task formulation has two advantages . 1 ) Instead of treating the question as a whole for answer event prediction , which is standard for existing question answering models , OTRE ’ s explicit decomposition of the question provides an opportunity to construct a context-agnostic representation of the temporal relations between events , which is more effective at leveraging the limited supervision signal available to generalize to unseen contexts and events . 2 ) It also allows us to more effectively and explicitly model the difference in temporal relations that are expressed similarly in natural language with a contrastive loss function , including ones that are mutually exclusive ( e.g. , “ happened before ” and “ happened after ” ) and ones that are different in a more nuanced manner ( e.g. , “ happened before ” and “ happened right before ” ) . This helps improve the model ’ s coherence in its answers to 1 . Note that TemporalRelations ( C ) does not explicitly exist , and is used here for notational purposes only . The task of open temporal relation extraction itself is to learn to find such triples from annotated question answering data . questions that are centered around the same question event . The OTRE decomposition allows us to propagate the difference defined by different answer sets effectively to open temporal relation representations , which further helps with generalization . 3.2 The OTR-QA Model . In this section , we introduce our question answering model based on open temporal relation extraction , OTR-QA . Given an input question Q , it is mainly processed in two steps to arrive at the final answer : 1 ) Question decomposition and context encoder ; 2 ) Answer event prediction from these learned representations that identifies temporal relation triples from the context . During training , we further regularize the model with two auxiliary loss functions : one for event detection , and the other for contrasting questions that share the same eq but have different answers .

Summary: Authors introduce a temporal question answering (TQA) model called OTR-QA, which decomposes a given question into a question event and a temporal relation. Their main contribution is to learn open temporal relations that do not require a predefined set of relations. The architecture is simple and trained with multiple hyperplanes. Answer event prediction loss is further assisted by two auxiliary losses (event detection loss and contrastive loss). Problem definition is straightforward (i.e., extract all the answer entities given the question entity and the relation) and the model is evaluated on TORQUE (Ning et al., 2020). Compared to the RoBERTa-large baseline by Ning et al., 2020, OTR-QA achieves +1.1% F1, -0.2% EM, and +3% C score (consistency). Authors also show that their model is more data-efficient and modeling the open temporal relations is indeed the key to achieve strong performance.