KnowComp at SemEval-2023 Task 7: Fine-tuning Pre-trained Language Models for Clinical Trial Entailment Identification
Year: 2023

Abstract:
In this paper, we present our system for the textual entailment identification task as a subtask of the SemEval-2023 Task 7: Multi-evidence Natural Language Inference for Clinical Trial Data. The entailment identification task aims to determine whether a medical statement affirms a valid entailment given a clinical trial premise or forms a contradiction with it. Since the task is inherently a text classification task, we propose a system that performs binary classification given a statement and its associated clinical trial. Our proposed system leverages a human-defined prompt to aggregate the information contained in the statement, section name, and clinical trials. Pre-trained language models are then finetuned on the prompted input sentences to learn to discriminate the inferential relation between the statement and clinical trial. To validate our system, we conduct extensive experiments with a wide variety of pre-trained language models. Our best system is built on DeBERTa-v3-large, which achieves an F1 score of 0.764 and secures the fifth rank in the official leaderboard. Further analysis indicates that leveraging our designed prompt is effective, and our model suffers from a low recall. Our code and pre-trained models are available at https://github.com/HKUSTKnowComp/NLI4CT.

heading: 1 Introduction
text: Recently, the proliferation of Clinical Trial Reports (CTRs) provides a large-scale reference base of scientific and factual knowledge for medical practitioners to make evidence-based clinical diagnoses (Zlabinger et al., 2020). However, it also posits the challenge that clinical practitioners cannot memorize all current literature in order to provide up-to-date personalized evidence-based medical care (DeYoung et al., 2020). With the recent advances in Natural Language Processing (NLP)
∗ Equal Contribution
systems, multiple language models pre-trained in the medical domain have been proposed to tackle medical NLP tasks efficiently (Rasmy et al., 2021; Lewis et al., 2020b; Liu et al., 2022; Kanakarajan et al., 2021). This makes NLP systems more practical and feasible to support the large-scale interpretation and retrieval of medical evidence (Marshall et al., 2020; Molinet et al., 2022; Pradeep et al., 2022; Yasunaga et al., 2022). Though current works study the application of state-of-the-art NLP techniques in the medical domain extensively, evaluation benchmarks are still not comprehensive enough. In this manner, Jullien et al. (2023) propose the Multi-evidence Natural Language Inference for Clinical Trial Data (NLI4CT) task by constructing an effective evaluation benchmark based on a collection of breast cancer CTRs1, statements, and explanations. All the collected data are in English, and the labels are annotated by domain experts. Specifically, the annotated statements make some specific types of claims about the informa-
1Extracted from https://clinicaltrials.gov/ct2/home
1
tion contained in one of the sections in the CTR premise that can be either focusing on one CTR only or comparing a pair of CTRs. The collected benchmark is associated with two proposed subtasks, which address both discrimination and retrieval problems. In this paper, we focus on the first task, textual entailment identification, which aims to determine the inferential relation between a medical statement and the collected clinical trials. Given a statement and CTR pair, the objective is to predict whether the statement affirms a valid entailment of the CTR or forms a contradiction with it, as shown in Figure 1. A system with superior performance on this task can retrieve medical entailment in real-life CTRs and provide clinical practitioners with accurate predictions of treatment outcomes (Katsimpras and Paliouras, 2022), which in turn aids in diagnosis and treatment (Zhang et al., 2020b).
With the recent advancement in Pre-Trained Langauge Models (PTLMs) on text classification tasks (Howard and Ruder, 2018; Wang et al., 2023b), we propose a simple yet effective system that is purely built based on fine-tuning PTLM. By using a carefully designed textual prompt, we aggregate the clinical statement, the claimed section name, and related CTR premises together for the models to learn cohesively. The models are asked to perform binary classification given a prompted input sentence, which stands for discriminating whether the statement claims an entailment or forms a contradiction. To validate the effectiveness of our proposed system, we evaluate a wide variety of PTLMs on both the validation and testing sets and study the ablation of our designed prompt.
Extensive experiment results demonstrate that our system maximally achieves an F1 score of 0.764 and ranks fifth on the official leaderboard. Specifically, DeBERTa-v3 (He et al., 2023b) achieves the best performance and is much more performant than other PTLMs, which highlights its strong language modeling capability. While the large version of other PTLMs may be hard to converge and cannot surpass their respective base version, DeBERTa-v3-large significantly outperforms its smaller version. Further analysis results demonstrate that dropping our designed prompt leads to performance drops, which demonstrates the effectiveness of our designed prompt. However, we observe that our model can only achieve a recall score of 0.772, which ranks thirteenth on
the official leaderboard and is a key bottleneck that prevents our system from achieving a higher F1 score. Future works should focus on improving the models’ robustness in discriminating true positive examples. Our work studies the performance of various PTLMs on medical multi-evidence natural language reasoning. The experimental results can greatly help clinical practitioners to provide personalized care. We thus make our data, code, and finetuned models publicly available2 for future contributions.

heading: 2 Problem Definition
text: In the textual entailment identification task, each input data contains three components: a medical statement, a section name indicating which section the statement claims about, and one or two CTR records that serve as the evidence to verify the statement. Specifically, if the statement only makes claims about one certain trial, then only the corresponding trial will be used as input data. On the other hand, if the statement claims a comparison between a primary trial and a second trial, both CTRs need to be considered and are provided to the model as input information. Consequently, the textual entailment identification task aims to determine the inferential relation between the medical statement and the associated section(s) in the claimed CTR(s). There are two possible inferential relations for each statement: entailment and contradiction. Models are expected to predict whether each statement affirms an entailment or forms a contradiction given the associated section from the claimed CTR(s).
Formally, denote the medical statement as t, the section name as st, the inferential relation as rt, and one CTR as C = {Cs1, Cs2, Cs3, ..., Cs|C|}. If two CTRs are involved, they are denoted as C1 and C2. The models will be given (t, st, Cst) or (t, st, C 1 st , C 2 st) as input and required to predict yt that will be compared against rt. We use the dataset provided by Jullien et al. 2https://github.com/HKUST-KnowComp/NLI4CT
(2023) to study this task. In the dataset, a total of 2,400 statements are split evenly across the different sections and classes. The statements and evidence are generated by clinical domain experts, clinical trial organizers, and research oncologists from the Cancer Research UK Manchester Institute and the Digital Experimental Cancer Medicine Team. Each Clinical Trial Report (CTR) consists of 4 sections: Eligibility criteria, Intervention, Results, and Adverse events. It may contain 1-2 patient groups, which may receive different treatments or have different baseline characteristics. Detailed statistics are provided in Table 1. More explanations regarding the CTRs and sections are provided in Appendix A.

heading: 3 System Overview
text: In this section, we introduce our proposed system. A general sketch is presented in Figure 2.

heading: 3.1 Prompt Design
text: The first step is to aggregate different input components into a whole sentence before performing further classification. Following the definition in Section 2, we extract t, st, C1st , C 2 st and use a discrete prompt to concatenate them together. For each section in a clinical trial, we use commas to link each section’s evidence into a paragraph to incorporate the prompt. To efficiently separate different components instead of introducing external natural language guidance, we leverage several predefined separator tokens, denoted as [SEP], to perform the isolation. The exact prompt used for training can then be denoted as “t [SEP] st [SEP] C1st [SEP] C 2 st”. Other pre-defined special tokens,
such as [CLS], [BOS], [EOS], are appended at their appropriate positions. The prompted sentences are further passed as input to the language models.

heading: 3.2 Encoder Model Selection
text: We experiment with a wide collection of popular transformer-based (Vaswani et al., 2017) pretrained language models as text encoders to obtain the embedding of the prompted input sentences.
BERT (Devlin et al., 2019): BERT is the very first bidirectional language model that is purely based on transformer architecture. It is pre-trained by using Masked Language Modeling (MLM) and Next-Sentence Prediction (NSP) objectives.
BioClinical-BERT (Alsentzer et al., 2019): BioClinical-BERT is a BERT model that is pretrained using the medical notes from a database containing electronic health records from ICU patients at the Beth Israel Hospital in Boston. It is considered to be a domain-specific language model that is potentially effective in clinical NLP tasks.
ALBERT (Lan et al., 2020): ALBERT is a variant of BERT that leverages two parameter reduction techniques, factorized embedding parameterization and cross-layer parameter sharing, to improve its parameter-efficiency. A sentence order prediction loss during pretraining is also introduced to replace the original NSP objective.
BART (Lewis et al., 2020a): BART is an encoder-decoder model that uses a bidirectional encoder and an autoregressive decoder to perform sequence-to-sequence modeling. It is pre-trained with the objectives of corrupting text with an arbitrary noising function and learning a model to reconstruct the original text. It has been shown to
be effective on both generative and discriminative NLP tasks.
RoBERTa (Liu et al., 2019): RoBERTa is a variant of BERT that is obtained by using an improved recipe for training BERT models. The training takes advantage of dynamic masking, full sentences without NSP loss, large mini-batches, and a large byte-level byte-pair encoding strategy. As a result, the performance of RoBERTa is significantly improved compared with the traditional BERT model.
ELECTRA (Clark et al., 2020): ELECTRA introduces a new way of pretraining MLM language models by corrupting the input by replacing some tokens with plausible alternatives sampled from a small generator network instead of performing the traditional input masking. In addition, the model is asked to predict whether each token in the corrupted input was replaced by a generator sample instead of predicting the original identity of the masked tokens. It is much more efficient than previous pretraining approaches and can achieve comparable results.
DeBERTa (He et al., 2021, 2023b): DeBERTa is the current state-of-the-art language model that improves the BERT and its variants by using disentangled attention and enhanced mask decoder. Meanwhile, the training efficiency of DeBERTa is also improved using ELECTRA-Style pre-training
with gradient disentangled embedding sharing. Performances on downstream natural language understanding tasks are significantly improved compared with previously introduced models.
GPT2 (Radford et al., 2019; Brown et al., 2020): GPT2 is a generative language model that is pretrained using a Causal Language Modeling (CLM) objective. In addition, the model takes a batch of sequences of the continuous text of a certain length as inputs, and the respective targets are the same sequence with one token shifted to the right. It also uses a masking mechanism to ensure that the prediction of a specific token is only based on previous existing tokens instead of future tokens. Such autoregressive training enables GPT2 to be extremely powerful in generating texts.
Considering the unique design and training objective function of each model, we adopt different extraction strategies for different models to extract the representation embedding of the input sentence from the token embeddings encoded by the language models. Specifically, for language models with a masked language modeling objective, including BERT, ALBERT, RoBERTa, DeBERTa, and ELECTRA, the embedding of the [CLS] token is used as the representation vector. While for autoregressive generative models such as BART and GPT2, the embedding of the last token in the
decoder is used as the representation of the input sequence. Such a strategy best utilizes the information provided by the encoder PTLM (Fang et al., 2022).
Two linear layers are connected after the encoder language model to perform binary classification on the representation vectors. Tanh function is used as the activation function (Nwankpa et al., 2018) and two dropout (Srivastava et al., 2014) layers are added appropriately to avoid overfitting.

heading: 3.3 Model Training
text: Denote the prompted input sentence as xi with |xi| tokens, the inferential relation label as yi. All models, denoted as θ, are trained using a standard cross-entropy loss, as shown in Equation 1. The predictions of xi, denoted as θ(xi), will be used to compute the loss against the truth label yi.
L(xi, θ) = − |x|∑
i=1
yi log(θ(xi)) (1)

heading: 4 Experimental setup
text: The data split for training, validation, and testing sets follows the original split as released by Jullien et al. (2023). Our system is built upon the Huggingface Transformers3 Library (Wolf et al., 2020). pre-trained tokenizers and language models are applied directly for further finetuning. The training and evaluation codes are mainly adapted from Fang et al. (2021b,a, 2023). We use a default learning rate of 5e-6 with a batch size of 4 to train the models. An AdamW (Loshchilov and Hutter, 2019) optimizer is used to update the parameters. The max sequence length for the tokenizer is set to 512, which is the most common longest input length for PTLMs. The models are evaluated on the validation every 10 steps by using precision, recall, and F1 scores (Powers, 2020). Early stopping is used where the best checkpoint is selected when the largest validation F1 score is achieved. The best checkpoint is further submitted to the CodaLab platform to acquire test set performances. We train our models with a sufficiently large number of epochs to ensure that underfitting does not occur. All experiments are repeated three times using different random seeds, and the average performance is reported. Four NVIDIA RTXA6000 (48G) and four NVIDIA RTX3090 (24G) graphical
3https://huggingface.co/docs/transformers
cards are used as the computational infrastructures. The number of parameters for every model is reported in Table 2. All testing set performances are acquired through submissions on the official CodaLab platform.

heading: 5 Results
text: The full results are shown in Table 2. We can observe that PTLMs with a Masked Language Modeling (MLM) objective generally can achieve satisfactory performance. The majority of them achieve an F1 score of above 0.6 on the test set. The GPT2 family, on the other hand, struggles with the task and can only achieve comparable performances when the parameter is over 700 million, which is nearly seven times more than other discriminative PTLMs. One possible reason is that autoregressive language modeling is not that competitive at classification tasks, and the GPT2 model cannot learn the negative samples well. Our best model is finetuned based on the DeBERTa-v3-large, which reaches a 0.764 F1 score and is significantly outperforming other models. This may be due to the fact that DeBERTa-v3-large possesses the largest number of parameters, and its disentangled parameter-sharing technique is effective for evidence-understanding tasks (He et al., 2023b). Such a result enables our model to secure a fifth rank on the official leaderboard and implies that advanced PTLM can be proficient in solving the textual entailment identification task.
We further study the ablation of our proposed prompt. DeBERTa-v3-large finetuned on two different input prompts are compared as baselines. Special Token stands for replacing the separator tokens [SEP] with natural language guidance such as “the evidence from CTR1 is”. Statement stands for using the statement only for prediction. The results, shown in Table 3, support our claim that leveraging predefined special tokens as separators and concatenating evidence after the statement is
effective. Meanwhile, our best model’s precision score is ranked fourth, and the recall score is ranked thirteenth. This indicates that the model is wrongly classifying many entailments as contradictions, which causes a relatively low recall and is considered a major limitation of our system. By further observing the errors made by our system, we find that challenging CTR-statement pairs involving complex or mathematical reasoning (Ferreira and Freitas, 2020) are often wrongly classified. In addition, due to the length of some CTR premises being much longer than the PTLM’s input max length, important pieces of evidence may be truncated and failed to provide guidance when making inferences on the statement. In our paper, we ignore this issue due to the fact that such cases infrequently occur through our manual inspection. However, positional encoding techniques such as relative position representation (Shaw et al., 2018) should be implemented to compensate for such information loss.
In addition, we foresee three lines of future works that can be addressed to enhance the system’s robustness further: (1) Leveraging evidence retrieval (Yadav et al., 2021; Conforti et al., 2020) or abstractive summarization (de Vargas Feijó and Moreira, 2023; Mao et al., 2022) to reduce the noise caused by irrelevant evidence and present the model with the most influential evidence. (2) Applying augmentation techniques (He et al., 2022; Wang et al., 2023a; Wu et al., 2022; Qin and Joty, 2022) and aligning medical inferences against large eventuality knowledge graphs (Zhang et al., 2022, 2020a; Röder et al., 2018) to mine inferential knowledge scalably and train the model more robustly. (3) Leveraging GPT3.5 (Brown et al., 2020; Ouyang et al., 2022), ChatGPT4 or other advanced large language models to generate or extract (He et al., 2023a) key evidence as they have shown strong capabilities in these tasks (Chan et al., 2023).

heading: 6 Conclusion
text: In this paper, we propose to finetune pre-trained language models to tackle the task of textual entailment identification as a solution to SemEval2023 Task 7. By using a discrete prompt to aggregate the medical statement and CTR premises, we maximally achieve an F1 score of 0.764 and ranked
4https://chat.openai.com/chat
fifth in the official leaderboard. Analysis experiments also prove the effectiveness of our proposed prompt. However, our system suffers from an excessively low recall rate. This indicates that the model misclassifies too many entailments as contradictions. Future work can focus on improving the ability of the model to identify entailments and improve its robustness. Our work demonstrates that applying a larger pre-trained language model can effectively identify clinical trial entailments, which can be applied to real-world scenarios. All of our code, data, and models are publicly available for future contributions.

heading: Acknowledgements
text: The authors would like to thank the committee of SemEval2023, the organizers of Task 7, and the anonymous reviewers. The authors of this paper were supported by the NSFC Fund (U20B2053) from the NSFC of China, the RIF (R6020-19 and R6021-20), and the GRF (16211520 and 16205322) from RGC of Hong Kong, the MHKJFS (MHP/001/19) from ITC of Hong Kong and the National Key R&D Program of China (2019YFE0198200) with special thanks to HKMAAC and CUSBLT, and the Jiangsu Province Science and Technology Collaboration Fund (BZ2021065). We also thank the support from the UGC Research Matching Grants (RMGS20EG01-D, RMGS20CR11, RMGS20CR12, RMGS20EG19, RMGS20EG21, RMGS23CR05, RMGS23EG08).

heading: A Dataset Description
text: In the dataset, the CTRs are collected from a database of privately and publicly funded clinical studies conducted around the world. The statements mainly claim on four sections of these CTRs: Eligibility Criteria: A set of conditions for patients to be allowed to take part in the clinical trial. Intervention: Information concerning the type, dosage, frequency, and duration of treatments being studied. Results: Number of participants in the trial, outcome measures, units, and the results. Adverse Events: These are signs and symptoms observed in patients during the clinical trial.

heading: Future Work
text: Future work can focus on improving the ability of the model to identify entailments and improve its robustness. Our work demonstrates that applying a larger pre-trained language model can effectively identify clinical trial entailments, which can be applied to real-world scenarios. All of our code, data, and models are publicly available for future contributions