Textual Entailment for Temporal Dependency Graph Parsing
Year: 2023

Abstract:
We explore temporal dependency graph (TDG) parsing in the clinical domain. We leverage existing annotations on the THYME dataset to semi-automatically construct a TDG corpus. Then we propose a new natural language inference (NLI) approach to TDG parsing, and evaluate it both on general domain TDGs from wikinews and the newly constructed clinical TDG corpus. We achieve competitive performance on general domain TDGs with a much simpler model than prior work. On the clinical TDGs, our method establishes the first result of TDG parsing on clinical data with 0.79/0.88 micro/macro F1. Our code is available at https://github.com/Jryao/thyme_tdg.

heading: 1 Introduction and Background
text: Temporal information extraction from text is an important part of natural language understanding. Many works have framed temporal relation extraction (RE) as the task of identifying temporal relations between pairs of events, or an event and a time expression (TIMEX3) (Pustejovsky et al., 2003a,b; Cassidy et al., 2014; Styler IV et al., 2014; Ning et al., 2018; Ballesteros et al., 2020; Lin et al., 2021). This pairwise framing can make it hard to decide when to annotate a temporal relation, and the resulting timelines are usually fragmented (Kolomiyets et al., 2012) as not all events or TIMEX3s are linked to each other. Heuristics are typically applied to constrain the search space of pairwise relations, both for annotators and machine learning models. For example, many annotation efforts have constrained temporal relations to adjacent sentences: TempEval (Verhagen et al., 2007, 2010; UzZaman et al., 2013), Clinical TempEval, (Bethard et al., 2015, 2016, 2017), and TimeBankDense (Cassidy et al., 2014).
A more principled approach is to model the temporal information in a document as a dependency tree structure (Kolomiyets et al., 2012; Zhang and
Xue, 2018b). This approach was extended by Yao et al. (2020) to temporal dependency graph (TDG) structure for a more comprehensive representation. An example is shown in Figure 1. With this approach, human annotators inspect each entity and find at most 2 reference times. A complete TDG can be constructed from these decisions. The automatic temporal RE task then becomes a parsing task: produce a TDG as output given a text as input. TDG datasets have been constructed for news articles (Yao et al., 2020) and contracts (Mathur et al., 2022). The current state-of-the-art (SoTA) TDG model (Mathur et al., 2022) reports 0.77 F1 score in the general domain, and 0.64 F1 on the contract dataset, showing the learnability of the TDG approach in those two domains.
In the current work, we make the following contributions:
• We bring TDGs to the clinical domain, by converting the pairwise annotations over the Mayo Clinic electronic health record (EHR) notes in the widely used THYME corpus (Styler IV et al., 2014) to TDGs.1
1Our THYME-TDG dataset will be available to the research community under the THYME data use agreement procedure.
191
• We develop an natural language inference (NLI)-based TDG parser that is much simpler than prior TDG parsers, yet achieves performance competitive with the state-of-the-art in the general domain. On the newly constructed clinical TDG dataset, this parser also achieves strong performance.
Our TDG parser is inspired by works applying NLI-based methods to other information extraction tasks, including relation extraction (Sainz et al., 2021) on the TACRED data set (Zhang et al., 2017), event argument extraction (Sainz et al., 2022) on the ACE (Walker et al., 2006) and WikiEvents (Li et al., 2021) datasets, and biomedical relation extraction (Xu et al., 2022).

heading: 2 Creating a Clinical TDG Corpus
text: A temporal dependency graph (TDG) is defined as a 4-tuple (T,E,M,L), where T is a set of TIMEX3s, E is a set of EVENTs, M is a set of pre-defined “meta” nodes (e.g. ROOT), and L is a set of edges. The definitions and guidelines2 of Yao et al. (2020) describe the steps to create a TDG from EVENTs and TIMEX3s: (1) For each TIMEX3 t, if t is locatable (i.e., t is not a QUANTIFIER, DURATION or SET), find its reference time expression (reference timex), otherwise assign no reference timex; (2) For each EVENT e, find its reference timex; and (3) For each EVENT e, find its reference event if there is one. Fig. 1 shows examples of such reference decisions, where the reference timex of “Friday” is “DocTime”, the reference timex of “sigmoidoscopy” is “DocTime”, and the reference event of “sigmoidoscopy” is “sees”.
We semi-automate this TDG construction process by leveraging the existing annotations over the THYME corpus (Styler IV et al., 2014). Our approach is visualized in Fig 2. First, we take all the EVENTs and TIMEX3s in the THYME corpus as the building blocks of the graph. In the following
2https://github.com/Jryao/temporal_dependency_ graphs_crowdsourcing
steps, we include as many TLINKs (temporal links) from the THYME corpus as possible to maintain the richness and informativeness of the THYME annotations. In some cases, we reverse the TLINK label (e.g. <e1 BEFORE e2> becomes <e2 AFTER e1>) to make the final graph structure simpler and the annotation process easier (see Appendix A.1).
Identifying the Reference Timex for a TIMEX3. TLINKs between two TIMEX3s are not annotated in the THYME corpus as the temporal relations between a pair of TIMEX3s can be inferred if their normalized values are available.3 For a locatable TIMEX3, we use the gold temporal operators annotations from the Semantically Compositional Annotation of Time Expressions (SCATE; Bethard and Parker, 2016) to get a TIMEX3-TIMEX3 relation by mapping temporal operators to temporal relations as shown in Table 1.

heading: Identifying the Reference Timex for an Event.
text: Given an event e, we choose the reference timex of e among the TIMEX3s linked to e in the original THYME corpus via TLINKs. If there is only one TIMEX3 temporally related to e, that TIMEX3 is automatically assigned as the reference timex of e. If there are multiple TIMEX3 temporally related to e, but only one TIMEX3 CONTAINS e, that TIMEX3 is automatically selected as the reference timex of e. Otherwise, the instance’s reference timex is manually annotated. If e is not TLINKed to any TIMEX3s, DocTime is selected
3https://clear.colorado.edu/compsem/documents/ THYME_guidelines.pdf
as its reference timex with the relation between an event and DocTime (DocTimeRel) as the label.

heading: Identifying the Reference Event for an Event.
text: To find the reference event for e among the EVENTs that are TLINKed to e, we define a label hierarchy4 based on the specificity of different temporal relations to facilitate consistent annotation:
BEGINS-ON, ENDS-ON >

heading: CONTAINS, CONTAINS-SUBEVENT >
text: BEFORE > OVERLAP, NOTED-ON
If the candidates have the same level of specificity, we choose the one closest to e in textual order.

heading: 3 Corpus Statistics
text: Following the Clinical TempEval tasks, we use the colon cancer set of the THYME+ corpus and the same data splits as described in Wright-Bettner et al. (2020). Our statistics on the Development (Dev) set show that there are on average 155 events per note. Since 91.5% of the TLINKs occur withinsection, and all the cross-section TLINKs are CONTAINS-SUBEVENT relations, we split each note into sections, drop the cross-section TLINKs, and create one TDG per section.
With our carefully designed conversion method, we automatically translated most of the pairwise annotations in the THYME+ corpus into TDGs.
4While the rest of the hierarchy reflects actual specificity, we do not claim that CONTAINS is more specific than BEFORE. We gave preference to CONTAINS here to capitalize on THYME’s narrative containers (Styler IV et al., 2014). This is also why we chose CONTAINS as the preferred TLINK type for selecting a reference timex.
There were only 72, 35 and 40 events in the Train, Dev and Test sets, respectively, which could not be automatically assigned reference times. Because of this small number, one domain expert manually annotated these missing TDG edges. Statistics of the final THYME-TDG corpus are in Table 2. The EVENT-EVENT rows show that more than 87% of the original event-event TLINKs are represented in the THYME-TDG corpus. Table 2 also shows the statistics of the publicly available TDG dataset (general-TDG; Yao et al., 2020).

heading: 4 TDG Parsing
text: The task of temporal dependency graph parsing is to find the parent node(s) for each child node, where the child node can be an EVENT or a TIMEX3. We cast the TDG parsing task as a textual-entailment (NLI) task. Given a child node xi and a list of candidate parent nodes {y1, y2, . . . , yi, . . . , yn}, our model first verbalizes the possible relations between <xi, yi> to generate hypotheses using a list of pre-defined templates (shown in Appendix A.3). For example, the verbalization for the BEFORE relation is: xi happened before yi. To get the premise, we concatenate the sentence that contains the child node and the one that contains the parent node.5 If yi is not the parent of xi, the relation label between xi and yi will be NOEDGE. We also add descriptions after the premise regarding the distance between two nodes, following previous works (Zhang and Xue, 2018a; Ross et al., 2020; Yao et al., 2020).6 Then, we run the NLI model to obtain the probability the premise entails/contradicts/is neutral to the hypothesis.
In the training stage, we finetune a pre-trained NLI model on the entailment data generated from the TDG training data. In a TDG, each event can have at most two parents: a reference timex, and a reference event. That is, most of the time, a candidate parent node is not the gold parent for the child, which means the relation between a child node xi and a candidate parent yi is NO-EDGE in most cases. To obtain a relatively balanced training set, we first divide the candidate parent nodes into two sets: A and B, where A contains the gold parent nodes for the current child node, and B contains the rest. Then, we randomly sample NE examples
5Only one sentence will be needed if the two nodes are in the same sentence.
6Our preliminary experiments show the additional features can increase the model performance by about 2%.
from A, and NC examples from B. For each instance sampled from A, we generate an entailment example using the gold label, and randomly sample NC incorrect labels to generate neutral examples. For each instance sampled from B, we generate one contradiction example (see Appendix A.3 for an example). In the inference stage, for each candidate parent node yi of a child node xi, we verbalize all possible relations between them and pick the candidate parent with the highest entailment probability as the final parent for xi.

heading: 5 Experimental Setup
text: We evaluate our model on the two TDG data sets: the general-TDG and THYME-TDG (clinicalTDG).7 When sampling the training data, we set NE to 1 and NC to 3. For general-TDG, to generate the reference timex candidates for each TIMEX3 or EVENT, we include all the TIMEX3s in the document; to generate reference event candidates for each EVENT, we include all the EVENTs from the beginning of the document to two sentences after the child node. For clinical-TDG, we include candidates in the window of 6 sentences before and 4 sentences after the child node. For both data sets, our candidate parent window setting covered > 99% of the cases.
For the general domain TDG parsing, we finetune the roberta-large-mnli (Liu et al., 2019; Williams et al., 2018) model via HuggingFace (Wolf et al., 2020) for 3 epochs. For the clinical domain TDG parsing, we finetune the PubMedBERTmnli-snli-scinli-scitail-mednli-stsb (Deka et al., 2022) model for 3 epochs. For model initialization, we experimented with 5 random seeds: {42, 52, 62, 72, 82}. See Appendix A.2 for other hyperparameters.
We use gold EVENTs and TIMEX3s as input for the TDG parsers. Parsed <child, relation, parent> triples are compared against gold triples to compute F1 scores. On the general-TDG, we report the average F1 scores across all documents (macroF) following previous practice (Zhang and Xue, 2018a; Yao et al., 2020; Ross et al., 2020; Mathur et al., 2022). On the clinical-TDG data set, we report both macro- and micro- F1 scores.
7The TDG dataset of 100 contracts used in Mathur et al. (2022) is not publicly available to the best of our knowledge.

heading: 6 Results and Discussion
text: Table 3 shows our general-TDG results. We compare our NLI-based TDG model with two existing supervised models trained for the TDG parsing task: the BERT-Ranking model (Ross et al., 2020)8 and the DocTime model (Mathur et al., 2022). Both our NLI-based TDG (best) and (average) models outperform the BERT-Ranking model by a large margin, suggesting the advantages of our NLI approach. The NLI-based TDG (average) model and BERT-Ranking model report the average scores of 5 runs, however it is unclear whether the DocTime results are the best or the average.
Compared to the DocTime model (Mathur et al., 2022), our NLI-based TDG model (best) achieved slightly lower but competitive performance, while being much simpler. The DocTime model contains 3 graph neural networks and relies on off-theshelf NLP tools including a co-reference resolution model, a dependency parser, a pre-trained model for sentence embeddings, and a document-level Rhetorical Structure Theory parser. Mathur et al. (2022) did not list exact tools or configurations (e.g., what is the model used for coreference resolution?) and the code is not publicly available, so it’s very hard to re-implement or apply this model to other data sets currently.
We evaluated our NLI-based TDG approach on the newly created clinical-TDG data set (Table 4). This is the first result with a graph algorithm on a clinical temporal relation dataset. Thus, our re-
8This model was not evaluated on the TDG data set by the authors, Mathur et al. (2022) ran the experiments on generalTDG and reported the results in their publication.
sults serve as a baseline for future research. Our NLI-based parser achieved promising results on the clinical-TDG data, showing both the utility of this dataset, and the generalizability of our TDG parser.

heading: 7 Conclusion
text: We explore TDG representation and parsing in the clinical domain. We convert the pairwise annotations over the Mayo Clinic EHR notes in the THYME corpus to TDGs semi-automatically. We then develop a NLI-based TDG parser that is much simpler than prior TDG parsers, yet achieves performance competitive with the SoTA in the general domain. On the clinical TDG data set, our parser also achieves strong performance, which can serve as a baseline for future research on clinical TDG parsing.

heading: Limitations
text: We finetuned pre-trained NLI models for TDG parsing. Both data sets we used were in English. To apply this model to other languages and to get the best results, pre-trained NLI models or NLI data sets might be required for the new language. Templates to verbalize the temporal relations in the new language are also required.
The clinical data set (i.e. THYME) we used in this work only contains EHRs from one institution: Mayo Clinic. Clinicians from different hospitals can have different writing style or use different templates when writing the notes. Future work should test the TDG representation and parsers on EHRs from other institutions, and EHRs of different patient populations.

heading: Acknowledgments
text: The research was supported by NIH (R01LM010090, R01LM01348). The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Library Of Medicine or the National Institutes of Health.

heading: A Appendix

heading: A.1 Annotation Details
text: Given a TLINK <event A, r, event B> in the THYME+ corpus, where event A is the Source (parent), event B is Target (child), and r is the label between them, we reverse the TLINK label in the following case: event B is the Target in multiple TLINKs, while event A is only TLINKed with event B.
In example 1 below, instead of looking for the reference event of colonoscopy between mass and bleeding, it’s easier to make colonoscopy the reference event of mass and bleeding by reversing those two TLINKs: <mass, NOTED-ON, colonoscopy> becomes <colonoscopy, NOTED-ON-INV, mass>, and <bleeding, NOTED-ON, colonoscopy> becomes <colonoscopy, NOTED-ON-INV, bleeding>, with “INV” indicating “inverse”. The subgraph representation for those 3 TLINKs is showed in Figure 3.
In example 2 below, it’s not clear which event among report, pathology, values and notes should
be the reference event of review as they all have the same temporal relation with review. However, if we reverse those TLINKs, then review will become the reference event of the 4 other events, as shown in Figure 4.
1. Review of the colonoscopy reports indicates that approximately 10-cm from the anal verge a 3- to 4-cm mass was seen with bleeding.
• <mass, NOTED-ON, colonoscopy> • <colonoscopy, CONTAINS, seen> • <bleeding, NOTED-ON, colonoscopy>
2. I have had the opportunity to review the operative report, surgical pathology, laboratory values, and notes.
• <report, BEFORE, review> • <pathology, BEFORE, review> • <values, BEFORE, review> • <notes, BEFORE, review>
A.2 Implementation Details For the general-TDG data set, we carried out a gridsearch of training epochs in {3, 4, 10, 20}, batch size in {16, 32, 64}, maximum sequence length in {64, 128}, learning rate in {1e-5, 2e-5, 3e-5}, and weight decay in {0.1, 0.2, 0.3}, final parameter settings are in bold.
For the clinical-TDG data set, we experimented with training epochs in {3, 4}, batch size in {16, 32}, learning rate in {1e-5, 2e-5, 3e-5}, and weight decay in {0.1, 0.2, 0.3}, final parameter settings are in bold.
Experiments were run on an NVIDIA Titan RTX GPU cluster of 7 nodes. It took 80 - 90 minutes to run one training epoch for both data sets.

heading: A.3 Templates
text: The templates we used to verbalize temporal relations are listed in Table 5 and Table 6.
We give a concrete example to show how we generate the NLI instances from our TDG data sets. Given an event ei, let {e1, e2, e3} be its candidate reference events, e2 be the gold reference event, and let BEFORE be the gold relation between e2 and ei. The following are the NLI instances we can generate for this example:
• Entailment: e2 happened before ei.
• Neutral: e2 happened at around the same time as ei.
• Contradiction: During e3, ei happened.
Both Entailment and Neutral examples are generated with the gold candidate event e2, the difference is that the Neutral instance has the wrong label that is randomly sampled from the label set. The Contradiction example is generated by randomly sampling an incorrect reference event from the candidates with a random label.
Please note that in both the training and inference stage, entity type constraints are applied when verbalizing a temporal relation. For example, in the general-TDG data set, “included” is only used for event-timex pairs. Therefore, when verbalizing event-event relations, the “included” label will be ignored.

heading: A.4 Features
text: The linguistic features we used are showed in Table 7.

heading: Future Work
text: Future work should test the TDG representation and parsers on EHRs from other institutions, and EHRs of different patient populations