YNU-HPCC at SemEval-2023 Task7: Multi-evidence Natural Language Inference for Clinical Trial Data based on a BioBERT Model
Year: 2023

Abstract:
This paper describes the system for the YNUHPCC team in subtask 1 of the SemEval2023 Task 7: Multi-evidence Natural Language Inference for Clinical Trial Data (NLI4CT). This task requires judging the textual entailment relationship between the given CTR and the statement annotated by the expert annotator. This system is based on the finetuned Bi-directional Encoder Representation from Transformers for Biomedical Text Mining (BioBERT) model with supervised contrastive learning and back translation. Supervised contrastive learning is to enhance the classification, and back translation is to enhance the training data. Our system achieved relatively good results on the competition’s official leaderboard. The code of this paper is available at https://github.com/facanhe/SemEval-2023Task7.

heading: 1 Introduction
text: With the massive increase of clinical trial reports (CTRs) publications, there are more than 10000 clinical trial reports for breast cancer at present. Therefore, it is not feasible for clinical practitioners to timely understand all current literature, so as to provide personalized evidence-based nursing (DeYoung et al., 2020). In this context, the application of natural language inference (NLI) can promote the development of large-scale interpretation and retrieval of medical evidence, and the successful development of the system can greatly promote us to combine the latest evidence to support personalized care (Sutton et al., 2020). To this end, the purpose of this task is to use clinical trial data to conduct multiple evidence natural language inference.
The task is based on a group of clinical trial reports of breast cancer, statements, explanations, and labels annotated by domain expert annotators. Each clinical trial report data of patients consists of four sections: (1) Eligibility criteria: a series of
conditions that allow patients to participate in the clinical trial. (2) Intervention: information about the type, dose, frequency, and duration of the treatment studied. (3) Results: number of participants in the trial, result measurement, unit, and results. (4) Adverse Events: physical signs and symptoms of patients observed during the clinical trial. An example of this task is shown in Table 1.
The SemEval-2023 shared task7 consists of two subtasks (Jullien et al., 2023):
• subtask 1: judgment of the reasoning relationship between the clinical trial report and the statement (contradiction or entailment);
• subtask 2: given a set of clinical trial reporting premises and statements, extract the supporting facts from the premises;
The main difficulty of subtask 1 is textual entailment. Textual entailment describes the reasoning relationship between two texts, in which text 1 is the premise and text 2 is the hypothesis. If the premise can infer the hypothesis, it is an entailment, otherwise, it is a contradiction. The task of textual entailment can also be regarded as the problem of text classification(Kong et al., 2022). The early task of textual entailment is based on the method of text similarity (Jijkoun et al., 2005) or text alignment (De Marneffe et al., 2008). With the development and expansion of deep learning, convolutional neural networks (CNN) (Kim, 2014) and recurrent neural networks (RNN) (Wang, 2018) have achieved good results in textual entailment. The long short-term memory (LSTM) (Rocktäschel et al., 2016) and application of the attention mechanism have also achieved better results than the CNN and RNN. However, since the introduction of the pre-trained model Bi-directional Encoder Representations from Transformers (BERT) (Devlin et al., 2019), it has raised the task of textual entailment to a new level.
664
To investigate the medical entailment relations, this study proposed to use the BioBERT model with supervised contrastive learning for subtask 1. Different from the conventional BERT encoder, BioBERT (Lee et al., 2020) was trained on Biomedical texts, thus can be more beneficial for the task of clinical text analysis. For the limitation of input length, different truncation strategies were conducted. In addition to the conventional crossentropy loss function, supervised contrastive learning (Gunel et al., 2020) was further introduced to enhance the classification. In each training batch, the samples with the same label are pulled together while the ones with different labels are pushed away in the semantic spaces for representation learning. On the original training data, back translation is used to enhance the data without changing the semantics, through this method, we doubled the training data. In addition, in order to obtain better results, the system explored another medical text processing model Bio_ClinicalBERT (Alsentzer et al., 2019), and the system also explored using the Longformer (Beltagy et al., 2020) model to deal with long text, but unfortunately, the experimental effect did not reach the expected effect, perhaps because the individual did not adjust the appropriate parameters.
Empirical experiments were conducted on the developing set to select the optimal solution for the final submission. The results show that BioBERT + Supervised Contrastive Learning + Back Translation has achieved the best performance of 0.665 in terms of the F1-score. The final submission for the test set has achieved 0.679 and ranked 12th place in subtask 1.
The remainder of this paper is organized as follows. Section 2 describes the model and method used in our system, Section 3 discusses the results of the experiments, and finally, the conclusions are drawn in Section 4.

heading: 2 System Description
text: This section will describe the architecture of the proposed model in detail. There are several com-
ponents in this section, including the tokenizer, the pre-trained model BioBERT, supervised contrastive learning, and back translation. The system model we proposed is shown in Figure 1.

heading: 2.1 Tokenizer
text: In many NLP tasks, the original text needs to be processed into digital data before it can be processed by computer.Thus, Tokenizer was applied to divide the text into words and convert it into unique coding. In the proposed model, the Bert tokenizer is used to build word vectors with a length of 512, which uses WordPiece to split the text into tokens. The final output X of the tokenizer is denoted as:
X = [CLS]x1x2...xn[SEP ]y1y2...ym[SEP ] (1)
where n and m represent the length of the first
Token
Embedding
Segment
Embedding
Position
Embedding
Embedding Output v
Input X
Figure 2: Embedding block
sentence and the second sentence; [CLS] special mark indicating the beginning of the text sequence; [SEP] indicates the separator between text sequences. The Tokenizer has different processing methods for sentences of different lengths. If the total length is less than 512, it uses zero to padding, and if the length is more than 512, it truncates.

heading: 2.2 BioBERT Model
text: BioBERT is a domain-specific BERT model based on the biomedical corpus. It uses the parameter weights of BERT-Base, and also uses PubMed Abstracts and PMC Full-text articles in the biomedical field as training data, which is much better than Bert in various biomedical text processing tasks. Therefore, the pre-trained model BioBERT was applied for subtask 1, it is based on the Transformer library 1. The structure of BioBERT is basically the same as BERT, which includes two core blocks: Embedded block and TransformerEncoder block. The main parameters of the model used in our experiment: 12 layers, 768 dimensions, 12 self-attention heads, and 109M total parameters.
Embedding block. After the original text is processed, it will first pass through this module. Its structure is shown in Figure 2. It consists of three blocks: Token embedding, which converts words into fixed dimension vectors; Segment embedding to distinguish the block of the current word; Position embedding indicates the absolute position of each word(Zhang et al., 2021).
Transformer Encoder block. This module is com1https://huggingface.co/dmis-lab/biobert-v1.1
posed of multi-layer TransformerEncoderLayer. Each TransformerEncoderLayer is composed of a multi-headed self-attention layer and a feedforward layer, denoted as:
Attention(Q,K, V ) = softmax(QK T√
dk )V
MultiHead(Q,K, V ) = Conact(head1, ..., headh)W 0
where headi = Attention(QW Q i ,KW K i , V W V i )
(2)
where Q, K, V is the embedding multiplication of tokens with three different initializing weight matrices and then initializing different Q, K, V by linear projection, the multi-head results are fused into a multi-head attention layer.
In the BioBERT model, the embedding input representation v is encoded by a multi-layer transformer, and the semantic association between each word in the sentence is fully learned with the help of the self-attention mechanism, and the contextual semantic representation h of the sentence is finally obtained. The embedding input representation v and the contextual semantic representation h are denoted as follows.
v = Inputrepresentation(X) h = BioBERT (v) (3)

heading: 2.3 Output Layer
text: The BERT model has two major pre-training tasks: mask language model (MLM) and next sentence prediction (NSP), and the text implication task usually uses the NSP method to predict, that is, use the hidden layer representation of [CLS] bits to predict the text classification(Ma et al., 2021). [CLS] is the first element of the input sequence, and its hidden layer representation is composed of h0, which represents the first component of h in context semantics. After obtaining the hidden layer representation h0 of [CLS] bit, the text label corresponding to the input text is predicted through the fully connected layer. For the probability distribution P of the text label, W0 ∈Rd×k represents the weight of the fully connected layer, h0 represents the offset of the fully connected layer, and k represents the number of classification labels. After obtaining the classification probability distribution P , calculate the loss with the real classification label y and learn the model weight. The calculation formula of the probability distribution is as follows.
P = Softmax(h0W 0 + b0) (4)

heading: 2.4 Methods
text: Supervised Contrastive Learning (SCL) takes samples of the same kind of labels as positive samples and different ones as negative samples. This idea is used in the fine-tuning part, in addition to using cross-entropy loss, we add contrastive learning loss on the classification task to make the same type of samples as close as possible and different types of samples as far as possible. For a multiclass classification problem with C classes, we use the training examples of size N, {xi, yi}i=1...N , Φ(·) ∈Rd represents an encoder that outputs l2 normalized final encoder hidden layer before softmax projection; yi,c is the label, P is the probability that the ith example belongs to class c, τ is the parameter that controls the separation of classes, and α is the scalar-weighted hyperparameter that sets the tuning for downstream tasks. The overall loss is then given in the following:
LCE = − 1N N∑ i=1 C∑ c=1 yi,c · logP LSCL = N∑ i=1 − 1 Nyi−1 N∑ j 1i ̸=j1yi=yj
· log exp(Φ(xi)·Φ(xj)/τ)∑N k=1 1i̸=k exp(Φ(xi)·Φ(xj)/τ)
L = (1− α)LCE + αLSCL
(5)

heading: 3 Experimental Results
text: Datasets. The NLI4CT shared task data set is composed of normalized JSON data, the size of training set train.json sorted by expert comments is 1700, the size of developing set dev.json is 200, the size of test set test.json is 500, and the size of the used clinical experiment report set CT json is 999. The data part of the expert mark mainly includes Type and Section_id, Primary_id, Secondary_id(comparison), Statement, Label, Primary_evidence_index, and Secondary_evidence_index(comparison). Type is used to indicate the test type (comparison/single); Section_id is used to indicate which part of the CTR is the statement commented by the expert. Primary_id, Secondary_id is used to indicate the CTR of a separate or comparative trial id; Primary_evidence_index, Secondary_evidence_index is used to indicate the evidence index that proves the label is correct.
Evaluation Metrics. The subtasks of the NLI4CT shared task are evaluated using the adopted standard evaluation indicators, including Precision, Re-
call, and Macro F1-score, the submissions of all teams are ranked according to F1-score. The metrics will be calculated as follows:
Precision = true positivestrue positives+false positives Recall = true positivestrue positives+false negatives F1-score = 2×Precision×RecallPrecision+Recall
(6)
Implementation Details. To facilitate model training and testing, the json data is firstly reorganized into traindata.csv, devdata.csv, and testdata.csv, which include uuid, label, statement, and premise. The premise is to extract the partial text of the corresponding CTR according to Section_id and type (e.g., if Section_id is results and type is single, extract the results text in CTR corresponding to Primary_id). The BERT model is firstly used as the baseline to implement the textual entailment task with processed training and developing sets. However, the result still has a lot of room for improvement. Perhaps because the text is about biomedicine, the model cannot fully capture the text semantics, so the biomedical domain model BioBERT and Bio_ClinicalBERT are used to complete this task After the use of supervised contrastive learning and back translation method and parameter tuning, we obtain better results than baseline.
However, due to the limitation of max position embeddings, the total size of words is limited to 512, including the [CLS] and [SEP]. However, the max length of the premise is 11227, and many of the hypothesis and premise texts add up to a total length of well over 512. In the above models, the length of texts over 512 are all directly truncated, which is simple but results in the loss of some semantics of the texts. In order to overcome this problem, the Longformer model is proposed for this task. However, due to GPU limitations and time constraints, the appropriate parameters were not adjusted to obtain the desired results, and there may be problems due to the wrong personal use method.
Hyper-parameters Fine-tuning. The warmup strategy is used to optimize the learning rate, which is a method for the learning rate mentioned in the ResNet (He et al., 2016) paper. In order to achieve the expected results, we adjusted different learning rates and batchsize to adapt to different models, the parameter tuning process is shown in the following
Figures 3 and 4, and set parameters in the final submitted results as follows: warmup steps is 500, weight decay is 0.01, the learning rate is 1.5e-5, train batchsize is 8, eval batchsize is 16, and epoch is 8.
Comparative Results and Discussion. The test is first carried out on the development set, whose size is 200. The true and predicted results are represented by the confusion matrix in Figures 5 and 6. Facing the different predicted results of BERT and BioBERT, it is clear that BioBERT performs better. Taking out individual examples and summarizing them in Table 2, we find that the reason for BioBERT’s better performance may be word segmentation. BERT used in this experiment is the uncased version, while the BioBERT is the cased version, and the difference is the case sensitivity. For example, the word INTERVENTION is divided by BioBERT and BERT Tokenizer respectively as [’ IN ’, ’##TE’, ’##R’, ’##VE’, ’##NT’, ’##ION’], [’ intervention ’], which could affect the predictions. Moreover, the medical terms in the text like the word Metastatic through the different tokenizers will be divided as [’Met’, ’##ast’, ’##atic’] and
[’meta’, ’##static’]. This has implications for the model to learn the semantics of the text in the next step.
The F1-score obtained from the experiments of several models and methods are summarized as shown in Table 3, and the result of the final submission is shown in Table 4. The F1-score of BERT is 0.541, the F1-score of BERT and Bio_ClinicalBERT using back translation are close, about 0.598, while the F1-score of BioBERT using back translation is 0.618. BioBERT performed well because it used biomedical corpus for training, while BERT only used general corpus. However, Bio_ClinicalBERT is trained using MIMIC III (a database containing electronic health records from ICU patients at the Beth Israel Hospital in Boston, MA.) on the basis of BioBERT. Its performance is inferior to that of BioBERT, possibly due to poor parameter adjustment, and the corpus is mostly about clinical surgery. The F1-score of BioBERT and BERT using supervised contrastive learning and back translation achieved 0.679 and 0.648 respectively. Obviously, the addition of supervised contrast learning makes the results of the model
better. The reason is that supervised contrastive learning brings similar labels closer, and separates different labels, which is more conducive to text classification. However, due to text length limitations, our experiment with Longformer did not work well, and there are still many areas for improvement.

heading: 4 Conclusion
text: In this paper, we present a system submitted in subtask 1 of the SemEval-2023 Task 7, which utilizes the pre-trained model BioBERT to adjust the official baseline model and uses the text classification task to complete the textual entailment task of CTRs. The experimental results show that our proposed system achieves good performance. In addition, in subtask 1, compared with the top-ranked team system, our system still has a lot of room for improvement. In future research, we hope to try other biomedical text-processing models or textlength processing methods to obtain better results.

heading: 5 Acknowledgement
text: This work was supported by the National Natural Science Foundation of China (NSFC) under Grant Nos. 61966038 and 62266051. The authors would like to thank the anonymous reviewers for their constructive comments.

heading: Future Work
text: In future research, we hope to try other biomedical text-processing models or textlength processing methods to obtain better results