FII SMART at SemEval 2023 Task7: Multi-evidence Natural Language Inference for Clinical Trial Data
Year: 2023

Abstract:
The “Multi-evidence Natural Language Inference for Clinical Trial Data” task at SemEval 2023 competition focuses on extracting essential information on clinical trial data, by posing two subtasks on textual entailment and evidence retrieval. In the context of SemEval, we present a comparison between a method based on the BioBERT model and a CNN model. The task is based on a collection of breast cancer Clinical Trial Reports (CTRs), statements, explanations, and labels annotated by domain expert annotators. We achieved F1 scores of 0.69 for determining the inference relation (entailment vs contradiction) between CTR statement pairs. The implementation of our system is made available via Github.

heading: 1 Introduction
text: Nowadays, investigation of clinical natural language processing tasks play a major role in the advancement of medical research, especially for innovation in evidence-based medicine (EBM) (Shivade et al., 2015). As reports of publications of Clinical Trial Reports (CTRs) continue to amass at rapid pace, it has become infeasible for clinical practitioners to remain constantly updated with the literature in order to provide personalized care (DeYoung et al., 2020).
Based on recent studies (Sutton et al., 2020), the legitimate research question is how we successfully connect the latest evidence to support personalized care? One direction is using textual entailment (Sammons et al., 2010; Cabrio and Manini, 2013), also known as Natural Language Inference (NLI) (Ravichander, A. et al., 2019, Silva et al., 2020), adapted for clinical texts. The relevant information from CTRs is structured
1https://github.com/volosincu/FII_Smart__Semeval2023
around the patient's problem format (Eriksen and Frandsen 2018): patient, intervention, comparison and outcome (PICO), formulation which makes clinical questions accessible and really helpful for further automated processing cause from human perspective the huge amount of data from randomized trials makes analysis endless and unavailing. Often, clinical results can be contradictory, with indeterminate findings due to variations in parameters (medication, age, location, duration, etc.) and creates an immense number of possible combinations between inclusion criteria, interventions, effects and outcomes, context in which the inference of evidence (Galashov, A. et al., 2019) was seen as a risky or too challenging area to tackle.
This perspective has known a great deal of progress in recent years due to developments of transformer models with fine-tuned variants like BioBERT (Bidirectional Encoder Representations from Transformers for Biomedical Text Mining) and SciBERT (a BERT model trained on scientific text) trained on medical corpora that facilitate rapid prototyping and experiments (Lee et al. 2019).
The rest of the paper is organized as follows: section 2 briefly presents studies related to textual entailment in clinical text, section 3 provides information about the system designed to determine the inference relation (entailment vs. contradiction) between CTR - statement pairs, while section 4 describes the experimental setups. Section 5 resumes the results of the conducted experiments, with their interpretations, followed by section 6 with the conclusions and further directions.

heading: 2 Background
text: This topic has attracted large attention in recent years, evidenced by increasing number of
212
scientific events (e.g., Workshop on Curative Power of MEdical DAta (MEDA 2017; 2018; 2020) (Cohen, K.B et al., 2020., Gifu, D. et al., 2019), Workshop on on Clinical Natural Language Processing (ClinicalNLP 2016; 2019; 2020; 2022) (Naumann, T et al., 2022; Rumshisky, A et al., 2020; Rumshisky, A et al., 2019; Rumshisky, A et al., 2016) and competitions.
Competitions such as SemEval-2023 Task 7: Multi-evidence Natural Language Inference for Clinical Trial Data are challenging, especially due to the data labelling problem, which is directly dependent on the automatic determination of the inference relation (entailment vs. contradiction) between CTR - statement pairs.
Different strategies are used in studies about clinical trial reports, from classical NLP techniques to transformer-based methods. The objective of this paper is to test if a transformer pretrained model on bio medical data can offer a competitive edge over the more classical approaches of NLP processing.
A consequent hypothesis, derived from initial analysis of the dataset, was that employing a traditional solution consisting of a CNN (convolutional neural network) model would lead to lesser or inconsistent results.
The dataset provided by organizers (Mael et al. 2023) for this competition consists of 999 breast cancer CTRs in English2, with statements, explanations and labels annotated by domain expert annotators. Each CTR3 may contain 1-2 patient groups, called cohorts or arms. These groups may receive different treatments or have different baseline characteristics. For task 1, each training instance contains 1-2 CTRs, a statement, a section marker, and an entailment/contradiction label. The annotations collected from reports of Randomized Control Trials (RCTs) represent the raw material and the input consumed by the system. The task entries provided by organizers for dev and test datasets are following the ICO intervention, comparison and outcomes format (DeYoung et al., 2020) and may refer to the same study or to two different studies.. For ease of reference, a demonstrative and trivial example would be to infer entailment between sentences
2 https://clinicaltrials.gov/ct2/home
chosen from a scenario like given a treatment A, a comparator B, and an outcome, the inference of entailment is to be established based on the statement and validity of the premises. There is an argument sustained by one or multiple premises with a “total of 2,400 statements split evenly across the different sections and classes” (Mael et al. 2023). The NLI4CT instances from where the prompts have been extracted is a collection of clinical trial reports (CTRs) formatted following the PICO standards consisting of:
● Clinical Trial Id
● Eligibility criteria
● Intervention
● Results
● Adverse effects.
Each clinical trial contains one or two patient groups (cohorts). The records and ICOs for development and testing are provided in JSON format (see Appendix D1). In the implementation, there are modules that handle data formatting before and after the model training and predictions.
3 Appendix D.1 A full display example of an NLI4CT document with annotation from a clinical trial record

heading: 3 System overview
text: The strategy we considered for this task represents a causal approach (see Figure 1) inspired from the architecture used in (Lee et al. 2019). The solution consists of a neural network model with multiple layers, the first of which is a pre-trained BioBert transformer extended with extra dense layers, detailed in Tables 1 and 2.
The given argument-premise pair forms a sparse input space, with medical terms, quantities and relations between them having a very subjective nature. The pretrain model is the first layer in the overall model architecture, having the role to reduce the dimensionality of the input argument-premises pairs4.
Semantic representation is a well-known method in the industry (Lee et al. 2019), but with the transformer model incorporated into the solution, we needed to leverage the capabilities and fine tune the mask and type ids settings. However, the results did not show improvements or relevant differences between the variants of the model where adaptations have been applied on the input mask of the transformer.
Attention mask is normally used to differentiate between the information containing tokens and the padding ones to predict the proper words based on local context. What if the attention would have been focused only on tokens with relevant information? The process to determine the relevant tokens presents itself as a challenging quest where discrete and extrinsic context come into play. This makes it a crucial step where more advanced work has to be done in future but for current implementation two examples are stated below. The first is a favorable one, because there is a similarity between the two pairs and sentences are smaller and have a local context well defined ECOG score smaller than 2.

heading: 3.1 Example
text: Study: NCT01125566 Argument, ECOG score < 2 is necessary to be eligible for the primary trial” Premise (Eligibility), „Eastern Cooperative Oncology Group (ECOG) score of 0 or 1”

heading: 3.2 Example
text: 4 The algorithm for processing the argument-premises pairs is presented in detail in as pseudocode in Appendix E.1
Study: NCT00328783
Argument „The adverse events section in the primary trial is empty” Premise (Adverse Events) „Total: 0/0”
In the case of the second example, it is challenging to determine if Total: 0/0 is strictly referring to all adverse effects or to some other aspects. Since the adverse section should be empty, then if it is not empty, it could mean that Total: 0/0 is pointing to other type of measurement. There are numerous cases that would require extra-processing and analysis before feeding a neural network and that can be added gradually as modules into the system. For a couple more examples that illustrate these kinds of situations see Appendix F1.
After input processing is complete, the semantic representation obtained from the BioBert encoder is sent further to extra dense layers to learn more data patterns. For the second solution we choose a standard CNN5. Although not a typical choice in text processing, since in the industry the convolutional networks are mostly known and employed in image processing there are instances and research where CNN are proven effective in causal inference problems (Ghasempour et al. 2023) reason why to be considered an adequate solution for textual inference task. For the preprocessing step, the same modules have been used, including
5 CNN notebook
the input structure of the transformer model, but in a different configuration of layers.

heading: 4 Experimental setups
text: Both solutions have been developed and trained in Google collab environment67. For the first experiment, using the transformer, in the medical domain we choose the BioBert pretrained model as best fit for the task and the dataset. The transformer model input tensor can accept up to 500 tokens, but this raises a resources challenge at memory allocation. In order to be able to train the system in reasonable time (minutes), the input has been pre-processed.
The solution was adapted so that the input is split into multiple instances, each with its own statement and premise pair, until we reached the parameter set for maximum sequence size for BioBert input. As extra processing, in case the size of premises was greater than max size, the premises were split in two or more entries. This helped us to mitigate the resource problem, leaving some questions open, especially regarding the possibility of improving accuracy by optimizing for larger inputs for models.
Some pairs argument - premises have lost context and the contradiction or entailment label did not match on all tested instances, but at the same time, having a more concise context might have helped in other regards. This is one area that
6 Transformer notebook
needs further attention, especially in deciding if and how the data can be better organized. For hyperparameters, the experiments have been run with multiple values from different ranges, empirically obtaining best results with size 16 batch and 15 epochs.
The practical sequence length for the BioBert input was between 100 and 290 tokens, values at which we could train the model within the limits of RAM maximum allocation (89Gb). Beyond that length the machine would go out of GPU memory and the model would crash.
The token sequence is built dynamically at runtime depending on the maximum size we set (ex. 290, 100). For each statement we search the premises in the CTR file and then merge arguments and premises (using [CLS] + [SEP] BERT). In the validation phase, after prediction, we iterate the results and search the entries that have been split and choose the final prediction based on majority voting.

heading: 5 Results
text: For the purposes of benchmarking and familiarizing participants with the task’s
7 CNN notebook
challenges, the organizers provided a starting kit that uses a simple TF-IDF algorithm. This baseline solution has achieved results of 48% F1 score, compared to our solution with the best result 69.9% F1 score for development data.
The complete history and settings (epochs, learning rate) are available in the notebook version history. It can be observed that training with chunked data with inputs size around 100 tokens issued predictions between 65-70%8. A plausible explanation could be that smaller sentences keep more concentrated semantic context, but this must be looked upon on a more detailed level since only some arguments could be validated by claims on semantic grounds.
Table 6 shows the results side by side for the 2 models with results. The choice of batch size and input size as references in table is based on the observation that these two parameters are most relevant for comparison and impact on results. The complete list of results shown in Appendix A.1 and B.1 contains the values after each retrain process for dev labeled dataset. One big and surprising advantage of the CNN model is that, in terms of training time and resources, it was much faster and required less computing power (RAM, CPU, GPU).

heading: 6 Conclusions
text: We demonstrated in this paper that a transformer pretrained model on bio medical for entailment relation task, does not necessarily give a competitive edge over a more classical approach, such as CNN. Compared with the baseline system, both solutions implemented by our team scored better results and give valuable insights into further investigations regarding how the architecture and model can evolve. Another area that can bring a lot of gains in the overall score is to extract essential information or
8 Detailed results and comparison tables in Appendix A.1 and Appendix B.1
relevant expressions from the statement and premises in order to match the maximum BioBert input vector and get more accurate semantic representation. It is possible that the inference task be more influenced by factors that are not necessarily dependent on medical context. We found that this is a challenging task (F-score = 0.69), with many open promising directions for further research.

heading: Future Work
text: This makes it a crucial step where more advanced work has to be done in future but for current implementation two examples are stated below. The first is a favorable one, because there is a similarity between the two pairs and sentences are smaller and have a local context well defined ECOG score smaller than 2