End-to-end clinical temporal information extraction with multi-head attention
Year: 2023

Abstract:
Understanding temporal relationships in text from electronic health records can be valuable for many important downstream clinical applications. Since Clinical TempEval 2017, there has been little work on end-to-end systems for temporal relation extraction, with most work focused on the setting where gold standard events and time expressions are given. In this work, we make use of a novel multi-headed attention mechanism on top of a pre-trained transformer encoder to allow the learning process to attend to multiple aspects of the contextualized embeddings. Our system achieves state of the art results on the THYME corpus by a wide margin, in both the in-domain and cross-domain settings.

text: The 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks, pages 313–319 July 13, 2023 ©2023 Association for Computational Linguistics

heading: 1 Introduction
text: Temporal information extraction is the task of discovering event mentions, time expressions, and relations between them that indicate their relative temporal ordering. The goal of these tasks is to place events on a timeline with as specific granularity as possible, to better understand topics that have strong temporal components. In the clinical setting, temporal information extraction can be applied to the text in electronic health records to create a timeline for a patient, enabling downstream applications that are heavily time-dependent (e.g., modeling the temporal ordering of problems and medication use to extract causally plausible candidates for adverse drug events). Recent datasets have been released as part of community shared tasks that enable the research community to make progress on this challenging suite of tasks (Styler IV et al., 2014; Sun
et al., 2013a; Wright-Bettner et al., 2019; Bethard et al., 2015, 2016, 2017). The THYME (1 and 2) and i2b2 datasets have led to progress in this area but these problems are far from solved.
Both the Clinical TempEvals (Bethard et al., 2015, 2016, 2017) and the 2012 i2b2 Challenge (Sun et al., 2013b) evaluated systems on an end-to-end basis – that is, given only raw text, participants needed to both extract relation arguments and relations between them. However, most systems attacked this setting with a pipeline approach, combining the best event and time extraction systems with a relation system. In recent general domain work, models for doing end-to-end relation extraction have been obtaining some success recently by building on pre-trained transformers in the BERT family (Devlin et al., 2019), using additional task-specific components that aggregate encoder information into relation predictions (Liu et al., 2020; Sui et al., 2020).
In this work, we extend Liu et al. (2020)’s model for the task of clinical temporal information extraction. Instead of using multi-headed attention for different relation categories, we use multiple heads to capture different aspects of the token representations. The additional layer allows the model to then integrate the signals to make the final relation predictions.
Our results show that this architecture can obtain state of the art performance for relation extraction on the Clinical Tempeval 2016 and 2017 tasks, which evaluated in-domain (colon cancer) and cross-domain (colon cancer → brain cancer) settings, respectively. Our analysis shows that, for this task, multi-headed attention is far superior to
313
single-headed attention with the same number of model parameters, suggesting that the multi-headed approach is valuable for representing different aspects of the input. In addition, we compare the trade-offs in modeling this as a multi-task learning setup, as opposed to using multiple fine-tuned classifiers that work separately, where the time expressions and events are found separately and used as the anchors for the relation classifier. Finally, we report results of this model on the THYME2 data, for which no current baseline results exist in the end-to-end task.

heading: 2 Background and Related Work
text: This work takes advantage of the THYME (Temporal History of Your Medical Events) corpus (Styler IV et al., 2014), and specifically the data released as part of the Clinical Tempeval shared tasks hosted at SemEval (Bethard et al., 2015, 2016, 2017). This English-language dataset uses colorectal and brain cancer notes from patients at the Mayo Clinic. Each patient is represented with 3-4 notes written around the time of cancer diagnosis, typically with two clinical notes and one additional radiology and/or pathology note. The first two Clinical TempEval’s evaluated on colon cancer only.
The third Clinical TempEval was a domain adaptation task where participants were given the colon cancer data and evaluated on the brain cancer test set. There were two settings, one with unsupervised domain adaptation (no access to brain training data), and another with a small amount of labeled brain cancer data. In practice, there was little unsupervised adaptation attempted, with one top system obtaining negative results with the technique of freezing embedding weights before training (to prevent some of the embeddings from updating and drifting away from target-domain-specific terms that might not be seen during training) (Tourille et al., 2017), and another also obtaining negative results by replacing rare terms in the training and test data with an UNK token (Leeuwenberg and Moens, 2017). In this work we use the THYME colon cancer data for training, the development set for tuning and model selection, and evaluate on both the colon cancer test set and brain cancer test set, in the unsupervised setting.
More recent work on THYME has focused on the easier relation classification setting, where gold standard time expressions and events are provided, and the system task is to decide which pairs of
temporal entities should be linked with temporal relations. The current best-performing system is from Lin et al. (2021). That paper’s main contribution was a continued pre-training technique for clinical text, where the masked language modeling task was modified to preferentially mask tokens that were part of temporal entities. The temporal relation extraction was modeled as a sentence classification task, where the gold standard arguments were marked by special tokens in the input, and a softmax layer off of the pre-trained transformer’s sentence representation token was used to predict the relation between the two marked tokens (including the None relation). Such an approach cannot be fairly compared (or even deployed) without building additional systems to do the event and time expression detection, so we do not directly compare to that result.
Other work in end-to-end relation extraction is also relevant to this work. The most relevant related work (Liu et al., 2020) used a BiLSTM encoder to get token representations, which fed into a multiheaded attention layer, where each attention head was used to score a different relation type. That work differs from ours in that it is applied only to “general-domain” tasks including the NYT (Riedel et al., 2010) and WebNLG datasets (Gardent et al., 2017), and treats outputs values from different attention heads as prediction scores for individual relation categories. Our work, in contrast, uses attention scores as features for a single downstream relation classifier (the CONTAINS relation), allowing the model to use different aspects of the representation to make classification decisions.
The current work also has connections to techniques for end-to-end relation extraction where the task was treated as a table filling task (Gupta et al., 2016), where each cell (i, j) in the table represents the relation between token i and token j.

heading: 3 Methods
text: Our model uses a multi-task learning setup built on top of a pre-trained transformer encoder, and specifically the PubmedBERT-base-uncasedabstract model (Gu et al., 2021). Time expressions and events are modeled as two independent BeginInside-Outside (BIO) tagging tasks, so that each contextualized token embedding in the sequence is used to make a classification decision for that token’s categorization as an event or time expression. This BIO tagging is implemented with softmax
layer at every token index over the B- or I- version of each of the time expression or event types. Relation classification is treated as a binary task applied to n2 token pairs to find relations, using a mechanism very similar to the self-attention mechanism already used extensively in transformers. Our attention-inspired classification model is an additional layer on top of the pre-trained transformer that takes in the contextualized token representations from the last layer of the transformer encoder.
Attention first came to wide use in encoderdecoder architectures (e.g. (Bahdanau et al., 2015; Cho et al., 2015)), as a way for a decoder to pick out relevant sub-components of the input (e.g., individual token representations for indeterminately long input sentences) during generation of outputs. Typically, it is implemented as a fullyconnected neural layer that learns a score for each sub-component (Kim et al., 2017) and uses the computed weighted average over the sub-components at each decoding step. Self-attention is a simple extension where the attention mechanism is used to update weights over a discrete set of elements at a higher layer l by applying attention over the same elements at a lower layer l − 1. Vaswani et al. (2017), in introducing the transformer architecture, showed that attention alone could be used both in encoding and decoding to learn hidden representations. Further, they introduced the idea of multi-headed attention, that is, the idea that several different attention distributions can be computed, as a way to focus on and combine multiple different aspects of the input representations.
The first stage in attention is projecting the input with Query, Key, and Value matrices, and then the attention is computed as:
Attention(Q,K, V ) = softmax (QKT )√
dk V
Multi-headed attention multiplies this out with distinct Q, K, and V projections for H attention heads (indexed by h), and then concatenates the outputs, with the sizes of the projection matrices scaled such that the total output dimensionality matches the input. For self-attention, the inputs to the Q, K, and V projections are all the same – the output from the previous layer.
The most important aspect of attention for our purposes is that the matrix product Sh = Qh(Kh)T in the numerator results in a matrix of size N × N for N tokens. In the work by Liu
et al. (2020), each relation type is aligned to one attention head, so the value at each position in the matrix represents the affinity score for the relation aligned to head h:
sh(i, j) = Sh(i, j)
In this work, we use H attention matrices to compute a single H × N × N tensor, where a softmax along the attention head dimension leads to a classification for each token pair into a single relation, the temporal narrative container relation:
pc(i, j) = softmax([S 0 i,j , S 1 i,j , · · ·SHi,j ])
The overall loss function combines time expression extraction, event extraction, and relation classification, all of which use versions of cross-entropy loss. The overall loss function simply sums the three, with a weight on the relation loss that we tune on the development set to maximize F1 score for the relation classification task.
Ltotal = Ltimex + Levent + αrel ∗ Lrel

heading: 4 Evaluation
text: We evaluate on the THYME colon and brain corpora, tuning hyperparameters on the colon dev set and reporting results of the narrative container extraction task on the colon and brain test sets.1 Hyperparameters we tuned over included fine tuning learning rate, batch size, classifier layer (which layer the classification head is connected to), relation task weight, number of classifier attention heads, and dimensionality of attention heads. For all reported results we use the official SemEval scoring tool, which performs transitive closure for the relations (UzZaman and Allen, 2011). Table 1 shows the primary outcomes, the precision, recall, and F1 scores for the narrative container relation. We compare against the best-performing system from the relevant Clinical TempEval shared task – for colon cancer it was a system from UTHealth (Lee et al., 2016) and for brain cancer it was a system from GUIR (MacAvaney et al., 2017).
While the system we describe is multitask in nature, it does not have explicit interactions between the sequence tagging models and the relation predictor. We first find predicted relations between tokens and work backwards to find the argument span
1SemEval 2016 and 2017 also used only the narrative container extraction task due to its superior inter-annotator agreement.
that exists at each token to make a relation prediction. This independence allows us to investigate the importance of the quality of the extraction of those entities, by using alternative methods, including an oracle, in the second step of finding the arguments for predicted relations. The row labeled + Gold args shows the performance with oracle-based arguments taken from the gold standard, to show us an upper bound on how well relation extraction would work with perfect argument extraction. The row labeled - MTL shows the performance if we use separately tuned event and time expression extraction systems, since multi-task learning sometimes can cause degraded performance of independent components.
The E2E-MHA system obtains the best performance on both corpora. For the colon cancer corpus, precision is 14 points higher than the best performing Clinical TempEval system, while recall is within one point, giving a 5-point increase in F1. For the brain cancer corpus, precision decreases by 11 but the increase in recall results in a small 2-point increase in F1.
The next result we report is on the importance of multi-headed attention. Our hypothesis is that using multiple attention heads allows the model to more easily learn to focus on multiple facets of an instance. An alternative hypothesis is that the gain comes from using a lot more parameters than an architecture that uses one attention head per relation (e.g., (Liu et al., 2020)). To test this, we can compare against a similar model that has fewer attention heads but more feature dimensions per head, holding the total number of model parameters constant. For this experiment, we report colon development set validation scores directly as measured by scikit-learn (i.e., not using Clinical TempEval scorer, so it does not include temporal closure), as we tuned this parameter before running final tests.
Table 2 shows the results of this experiment. We found there was a sweet spot with 256 attention heads of size 64 each, where a similarly parameterized versions lose performance. As the number of heads decreases to <= 4, performance drops to an extent that finding good hyperparameter settings was not possible in our limited search.
Finally, we report the results when the same system is tuned, trained, and applied to the THYME2 corpus (Wright-Bettner et al., 2020). THYME2 introduced a new relation type called NOTEDON
which was used to indicate a test noting a finding, which had previously been covered by CONTAINS. This change improved annotator agreement and removed logical inconsistencies that were created by the previous schema. Another new relation type in THYME2 is CONTAINS-SUBEVENT, which was primarily used for cross-document annotations. For THYME2 we report results micro-averaged across all relations in Table 1, and break these results down by relation category in Appendix A. As far as we are aware, this is the first reported end-to-end temporal relation extraction result on THYME2.

heading: 5 Discussion and Conclusion
text: While this task remains challenging, with the highest obtained F1 scores 0.53 (in-domain) and 0.36 (out-of-domain), the combination of biomedically pre-trained transformers and a novel multi-headed attention classifier obtains results that improve the state of the art. Even in-domain, the task is fairly far from being solved.
Additional future work should also explore explicit domain adaptation, as out-of-domain perfor-
mance still suffers despite obtaining better performance than previous methods. Simple unsupervised methods like target domain continued pretraining (Gururangan et al., 2020) are likely to provide benefits on the brain cancer corpus, and future work should investigate this and other approaches. We did see performance improvements in preliminary work when switching from general domain pre-trained models (roberta-base Liu et al. (2019)) to a biomedically trained model, so it is possible that some of those potential gains are already incorporated by the more general language model.
The trained models developed for this work are available in the HuggingFace Models repository.2 The code for implementing and evaluating these models is available on GitHub.3 The code repository also contains a FastAPI-based demonstration server that sets up a REST server to convert text arguments into temporal relations in a simple JSON schema.

heading: Acknowledgements
text: Research reported in this publication was supported by the National Library Of Medicine of the National Institutes of Health under Award Numbers R01LM012973, R01LM012918, and R01LM010090, by the National Institue on Drug Abuse under Award Number R01DA051464, and by the National Institute Of Mental Health of the National Institutes of Health under Award Number R01MH126977. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

heading: A THYME2 Results by Relation Category

heading: Future Work
text: Additional future work should also explore explicit domain adaptation, as out-of-domain perfor-
mance still suffers despite obtaining better performance than previous methods. Simple unsupervised methods like target domain continued pretraining (Gururangan et al. , 2020) are likely to provide benefits on the brain cancer corpus, and future work should investigate this and other approaches. We did see performance improvements in preliminary work when switching from general domain pre-trained models (roberta-base Liu et al. (2019)) to a biomedically trained model, so it is possible that some of those potential gains are already incorporated by the more general language model. The trained models developed for this work are available in the HuggingFace Models repository. 2 The code for implementing and evaluating these models is available on GitHub. 3 The code repository also contains a FastAPI-based demonstration server that sets up a REST server to convert text arguments into temporal relations in a simple JSON schema