SSNSheerinKavitha at SemEval-2023 Task 7: Semantic rule based label prediction using TF-IDF and BM25 techniques
Year: 2023

Abstract:
The advancement in the healthcare sector assures improved diagnosis and supports appropriate decision making in medical domain. The medical domain data can be either radiology images or clinical data. The clinical data plays a major role in the healthcare sector by preventing and treating the health problem based on the evidence learned from the trials. This paper is related to multi-evidence natural language inference for clinical trial data analysis and its solution for the given subtasks (SemEval 2023 Task 7 NLI4CT). In subtask 1 of NLI4CT, the inference relationship (entailment or contradiction) between the Clinical Trial Reports (CTRs) statement pairs with respect to the Clinical Trial Data (CTD) statement are determined. In subtask 2 of NLI4CT, predicted label (inference relationship) are defined and justified using set of supporting facts extracted from the premises. The objective of this work is to derive the conclusion from premises (CTRs statement pairs) and extracting the supporting premises using proposed Semantic Rule based Clinical Data Analysis (SRCDA) approach. From the results, the proposed model attained an highest F1-score of 0.667 and 0.716 for subtasks 1 and 2 respectively. The novelty of this proposed approach includes, creation of External Knowledge Base (EKB) along with its suitable semantic rules based on the input statements.

heading: 1 Introduction
text: The Natural Language Processing (NLP) is one of the fastest growing sub-domain of Artificial Intelligence (AI) for medical domain (Basu et al., 2020). The contributions of NLP for medical domain are transcribing medical documents, clinical entity resolver and end-to-end drug discovery. To improvise the existing applications, research forums are conducting tasks for current requirements. One among them is SemEval 2023 and the task is multi-evidence Natural Language Inference (NLI) for clinical trial data (Jullien et al., 2023) and anal-
ysis. The motivation of this task is to help the clinical practitioner to stay updated on all current literature by providing an opportunity to support the large-scale interpretation and retrieval of medical evidence. To achieve this, NLI is applied on the clinical trial dataset (breast cancer Clinical Trial Reports (CTRs) and Clinical Trial Data (CTD)) to infer the medical evidence based on the relationship between premises and hypothesis. The clinical trial data analysis is divided into two subtasks namely, textual entailment and evidence retrieval.
In textual entailment (subtask 1), the hypothesis is derived from CTRs premises of four sections namely, intervention, eligibility, results and adverse events with respect to the CTD statement. Under each section there are three to eighty-two premises. In evidence retrieval (subtask 2), the anticipated premises which play a significant role in deriving the conclusion are retrieved.
The remaining part spans across the following subsections. The background for the textual entailment and evidence retrieval subtasks are discussed in Section 2. In Section 3, the system overview of the proposed work is explained. The experimental setup and implementation are explained in Section 4. A brief summary about the results, quantitative analysis and error analysis are given in Section 5 and conclusion is summarized at the end.

heading: 2 Background
text: The clinical trial dataset is common to both textual entailment and evidence retrieval subtasks of SemEval 2023 (Task 7). The dataset comprises Clinical Trial Data (CTD) (training set - 1700 samples, development set - 200 samples, test set - 500 samples ) and Clinical Trial Reports (CTRs) (CT JSON - 1000 samples). The training and development set of CTD consists of type (single or comparison), section_id (intervention, eligibility, results or adverse events), primary_id (map to CTRs ID), secondary_id (map
950
to CTRs ID), statement, label (entailment or contradiction), primary_evidence_index and secondary_evidence_index. The test set consists of type, section_id, primary_id, secondary_id and statement. If type is single, then secondary_id and secondary_evidence_index are inappropriate. The dataset structure can be better understood through the following example.
Let the primary_id, secondary_id and section_id of CTD be “NCT00856492”, “NCT00009945” and “Eligibility”, then the respective CTRs having the CTRs ID as "NCT00856492" and "NCT00009945" are selected from CT JSON folder. Within the selected files, the statements under the eligibility section are considered for the proposed work.
From the dataset analysis, the following keypoints are observed, (i). Every statement in the CTRs section need not be required to derive the conclusion. (ii). In case of comparison type, statements from both primary section and secondary section contribute equally in achieving the conclusion. (iii). For better result, (Thamer et al., 2020) stated that semantic level interpretation is required (for example, the clinical trial statement "Most participants in the secondary trial and the primary trial did not suffer from Enterocolitis" is equivalent to the statement “Enterocolitis haemorrhagic 1/167 (0.60%)” and “Enterocolitis haemorrhagic 0/167 (0.00%)” in the adverse event section (iv). External knowledge base (Wu et al., 2020) which maintains synonyms is required to develop better models. For example, the statement “Only patients with HER2 positive breast carcinoma are eligible for the primary trial” is equivalent to “HER-2 overexpressing breast cancer” in the eligibility section because “Carcinoma” is synonymous to “Cancer”.
After the analysis of the given clinical trial dataset, the Semantic Rule based Clinical Data Analysis (SRCDA) approach is designed and explained in the following section.

heading: 3 System Overview
text: The proposed SRCDA system for textual entailment and evidence retrieval is divided into three modules namely, SRCDA (Common to subtasks 1 and 2), SRCDA – textual entailment and SRCDA – evidence retrieval. The SRCDA first module consists of five submodules and each of the second and third module consists of two task specific submodules which is shown in Figure 1 for better understanding. The SRCDA module consists of prepro-
cessing, External Knowledge Base (EKB) creation, semantic rules formulation, feature vector generation by Term Frequency Inverse Document Frequency (TF-IDF) technique (Kim and M, 2019) and distance similarity metric (Cha, 2007) by Radial Basis Function Kernel (RBF-Kernel) modules. The two submodules specific to SRCDA – textual entailment are, Clinical Trial Reports (CTRs) section statements filtration and label prediction. The two submodules specific to the SRCDA – evidence retrieval are CTRs section statements ranking by Best Match 25 (BM25) techniques (Kamphuis et al., 2020) and, primary and secondary indices retrieval. These modules are discussed in detail in the following subsections and the process is explained in Algorithm 1.
The proposed SRCDA model uses EKB, semantic rules, TF-IDF techniques, RBF-Kernal distance metrics, BM25 technique because of the following reasons. They are: (i). The EKB collects the knowledge from multiple external sources for model creation; (ii). The semantic rules are defied to understand and analyse the statements by considering the context; (iii). The TF-IDF technique is used since it gives weights based on the uniqueness and relevance rather than frequency and hence more accurate representation among the overall polarity can be achieved; (iv). The RBF-Kernel is selected since it computes the similarity score between two statements based on their distance in the higher dimensional space; (v). The BM25 technique, supports Clinical Trial Report (CTR) statements length and term frequency saturation to rank the statements in the primary and/or secondary indices of CTR statements.

heading: 3.1 Semantic Rule based Clinical Data Analysis (SRCDA)
text: The proposed system design takes CTRs and Clinical Trial Data (CTD) as inputs, processes the data and generates the score value which will be used in subtasks 1 and 2.

heading: 3.1.1 Preprocessing
text: In preprocessing, the statements in the CTRs section and CTD are preprocessed as follows, (i). Removed stop words, punctuation, unnecessary space and preposition using Natural Language Toolkit (NLTK) (ii). Tokenized the statements as individual words (iii). Applied stemming and lemmatization methods on the tokenized input samples to get root words.

heading: 3.1.2 External Knowledge Base (EKB)
text: From the preprocessed unique tokens, the EKB is developed using external sources like PubMed, ImageCLEF (for medical terms) and two linguistic websites (for non-medical terms). This EKB consists of tokens (with respect to the vocabulary list) and its equivalent synonymous words are used for training the model. During testing phase, for the given CTD statement, tokens are masked with EKB synonymous words and hence equivalent CTR statements followed by conclusion is derived.

heading: 3.1.3 Semantic rules formulation
text: In this paper, four different semantic rules are framed with respect to the dataset. They are: (i). Negation equivalence rule; (ii). Double negation rule; (iii). Deductive reasoning rule; (iv). Condition based equivalence rule. The reason behind
using these four rules are: (i). In negation equivalence rule, the complement of CTD statement is equivalent to the negation of CTR statements; (ii). In double negation rule, the double negation of the CTD statement is equivalent to the CTR statements; (iii). In deductive reasoning rule, to deduct the CTR statements, the CTD statement is mapped with the corresponding universal truth statement; (iv). In condition based equivalence rule, based on the condition in CTD statement, the respective CTR statements are retrieved to derive the conclusion.
Negation equivalence rule
Let x be universal quantifier, defined on various premises as P1, P2, ...Pn (where n represents the number of premises) on which conjunction and disjunction is applied along with the negation con-
cludes Q. The CTD statement ∀x¬(P1(x) ∧ ¬(P2(x)) ∨ ...Pn(x)) ⇒ ¬Q(x) and CTR statement ∀P1(x) ∧ ¬(P2(x)) ∨ ...Pn(x) ⇒ Q(x) are equivalent if the value of x is identical in both case.
For example, the CTD statement "Pre and post menopausal women can enter the primary trial as long as they do not have prior hormone replacement theory" is equivalent to CTR statement "Pre or post menopausal women reporting use of hormone replacement therapy are not eligible for screening".
Let x be an axiom that represents women, W (x) be menopausal women, P (x) be premenopausal women, O(x) be post-menopausal women, PHRT(x) be prior hormone replacement therapy and E(x, PT) be the eligible patient for primary trial. (1) : ∀xW (x) ∧ (P (x) ∨ O(x)) ∧ ¬(PHRT (x)) ⇒ E(x, pt) (2) : ∀xW (x)∧(P (x)∨O(x))∧PHRT (x) ⇒ ¬E(x, pt) The negation of (1) gives (2), hence both statements are equivalent. As a result, this CTR statement is considered for deriving the conclusion.
Double negation rule Let x be universal quantifier, defined on various premises as P , applying double negation concludes P and it is represented as, ∀x¬(¬P (x)) ≡ P (x).
For example, the CTD statement "Patient who is not willing to sign and not willing to give informal consent are not considered for primary trial" is equivalent to the CTR statement "Patient who is willing to sign and give informal consent is considered for primary trial".
Let x be an axiom which represents patient, W (x) be willing patient, S(x, ic) patient willing to sign informal consent. ∀xP (x) ∧ ¬W (¬S(x, ic)) ≡ ∀xP (x) ∧
W (S(x, ic))
The double negation of CTD statement gives CTR statement and hence this CTR statement is selected for label prediction.
Deductive reasoning rule Let x be universal quantifier, defined on various premises as P concludes Q when both premises are universal truth statement. ∀xP (x) ≡ Q(x)|P (x), Q(x)ϵM where M is the universal truth statement For example, let the CTD statement be "Fiona’s sister who is 34 years old was diagnosed with a ductual carcinoma and therefore Fiona may be eli-
gible for primary trial" and CTR statement be "A first degree relative with breast cancer under the age of 60 are eligible for primary trial".
Let x be an axiom, S(x, fiona) be Fiona’s sister, HA(34) be an Fiona’s sister with age 34, H(x, d) be Fiona’s sister with disease ’d’, FDR(x, y) be an first degree relation, E(fiona, pt) be Fiona is eligible for primary trial, HA(age,< 60) be the FDR relation with age less than 60, E(y, pt) be eligible for primary trial. (1) : ∀xS(x, fiona) ∧ HA(34) ∧ H(x, ductualcarcinoma) ⇒ E(fiona, pt) (2) : ∀x, yFDR(x, y)∧H(x, breastcancer)∧ HA(age,< 60) ⇒ E(y, pt) The universal truth statements are Statement 1: "Sister is an first degree relation" Statement 2: "Ductual carcinoma is an early stage of breast cancer"
By the universal truth statements, it can been concluded that (2) follows (1) and hence both statements are equivalent. Hence, this CTR statement is also selected for label prediction.
Condition based equivalence rule Let x be universal quantifier, defined on various premises as P and Q along with some condition which is equivalent to the relevance condition in the CTR statement will give the consequence. ∀xP (x) ∧Q(r, condition) where r is a relation. From the deductive reasoning rule example, one of the condition in the CTD statement is ∀xP (x) ∧HA(34) and in CTR statement is ∀xP (x) ∧ HA(age,< 60). Both these statements are equivalent because, the age 34 is less than age 60, based on the condition it is equivalent and hence these statements fall under condition based equivalence rule. Hence this statement is considered for label prediction otherwise remaining CTR statement is evaluated based on the semantic rules.

heading: 3.1.4 Feature vector generation by TF-IDF technique
text: In Term Frequency Inverse Document Frequency (TF-IDF) computation, the preprocessed CTD statement and preprocessed semantically equivalent CTRs section statements are vectorised. The vectorization finds the relevant section statements based on the given statement, ranks it in the order of relevance and selects the top k section statements. The steps involved in this process are, find the frequency of token in the section statements, perform vectorization based on all possible synonyms words in
the EKB, assign less weight for stop words, find the number of section statements in which the tokens occurred, calculate top k section statements based on the number of tokens in each statement as given in Equations 1 to 3.
tfidf (s, c, C) = tf(s, c).idf(s, C) (1)
where,
tf(s, c) = log(1 + freq(s, C)) (2)
idf(s, C) = log( N
count(cϵC : sϵc) ) (3)
In the equation, s represents the processed CTD statement, c represents the processed semantically equivalent CTRs section statement and C is the corpus of all processed semantically equivalent CTRs section statements.

heading: 3.1.5 RBF-Kernel distance similarity metric
text: The RBF-Kernel distance similarity metric computes the score value of tf-idf vector matrices. It is similar to KNN algorithm but it has no space complexity problem because it stores only the support vector during training and not the entire dataset.
RBF −Kernel(vs, vc) = exp(− ∥vs, vc∥2
2σ2 ) (4)
Where vs, vc be the vectors computed for each CTD-CTRs statement pair. represents the variance of vs and vc. The equation 4 computes the similarity between vs and vc and ∥vs, vc∥2 is the Euclidean (L2 - norm) distance between vs and vc. Therefore, RBF-Kernel score is computed for each CTD-CTRs statement pairs.
If there are n CTD-CTRs statement pairs then there will be an n RBF-Kernel score for the particular statement.

heading: 3.2 Subtask 1
text: In subtask 1, the labels are predicted as given in Function 1 based on the critical score value computed from the distance similarity metrics for CTDCTRs statement pairs,

heading: 3.2.1 Filter CTRs section statements
text: The CTRs section statements are filtered based on the critical score value and it is fixed as 0.80, since the contribution of significant words derives the conclusion. If the score value of the particular statement pairs is greater than the critical score, then the respective CTRs statement is considered.

heading: 3.2.2 Predict labels
text: The label (hypothesis) is derived from the filtered CTRs statements (premises) by applying conjunction and disjunction on the set of premises. The hypothesis may be “contradiction” or “entailment”.

heading: 3.3 Subtask 2
text: In subtask 2, the statements (premises) which contribute to the answer prediction are retrieved by BM25 techniques as shown in Function 2.

heading: 3.3.1 BM25 technique
text: Based on the label (contradiction or entailment), the respective premises which contributes to the label prediction are extracted for the CTRs statements. In BM25 technique, the set of CTRs statements from both primary and secondary indices are ranked based on the CTD statement and labeled as given in Equations 5, 6 and 7.
BM25(si) = n∑
i
IDF (si) ∗ S (5)
where,
S = f(si, c) ∗ (m1 + 1)
f(si, c) ∗m1 ∗ (1− b+ b ∗ MN ) (6)
IDF = ln(1+ CTRstatemnetcount− f(si + 0.5)
f(si) + 0.5 )
(7) Where si is the statement, C is the set of CTRs statements, m1 is the term frequency saturation characteristics, b is set as 0.75 ( if b is bigger than the effects of the length of the document compared to the average length can be improved), M be each CTR statement length and N be an average of each CTR statement length.

heading: 3.3.2 Retrieve primary and secondary indices
text: Based on the rank value from BM25 technique and the score value from RBF-kernel similarity distance metrics, the primary and the secondary trial statements are extracted with the help of critical score. From the primary and secondary trial statements, the respective indices are retrieved as primary_index and secondary_index.

heading: 4 Experimental Setup
text: The system requirement for the proposed approach includes, (i). Hardware requirement – Intel i5 processor with NVIDIA graphics card, 4800M
at 4.3GHZ clock speed, 16GB RAM, Graphical Processing Unit and 2TB disk space (ii). Software requirement - Ubuntu 20.04 operating system, Python 3.7 package with required libraries like tensorflow, torch, sklearn, json, numpy, nltk, pickle, pandas, rank_bm25, etc.,
The Clinical Trial Data (CTD) in clinical trial dataset is divided into training (training and development) set and test set. Both training and test set uses files in the CT JSON folder for utilizing statements in Clinical Trial Reports (CTRs) section. The proposed model is developed as given in the system design for two subtasks for the training set. In the testing phase, test set is given as input to the proposed model to predict label (subtask 1) and, retrieve primary and secondary section indices (subtask 2). The effectiveness of the proposed model are evaluated by comparing the predicted output with the actual answer using performance metrics. The different performance metrics used for analysing the results are F1-score, recall and precision. In this, the F1-score measures the model accuracy with respect to the dataset, recall measures the number of correctly identified samples by
the model to that of labels, and precision identifies the frequency in which a model is correct while predicting the positive class.

heading: 5 Results
text: The performance of the proposed model for different runs are evaluated using suitable quantitative metrics and it is given in Table 1. Even though we have submitted 14 and 38 different combination of runs in two subtasks, some of them are discussed for quantitative and error analysis. In subtask 1, the proposed model is analysed for five different combinations of techniques, such as: (i). TF-IDF followed by Euclidean distance similarity, critical score value is fixed as 0.80 (ii). Same as (i), but the critical score value is reduced as 0.50 (iii). TF-IDF followed by cosine distance similarity (iv). Same as (i), but Radial Basis Function Kernel (RBF-Kernel) is used instead of Euclidean distance similarity (v). Preprocessing, TF-IDF followed by RBF-Kernel for critical score value of 0.72. From the results, it has been inferred that data preprocessing, role of External Knowledge Base (EKB), RBF-Kernel
selection and suitable critical score gives better performance than the other combinations.
For sub-task 2, the proposed model is analysed for five different combination of runs (four runs from prior submission and one runs from postcompetition), such as: (i). BM25 technique to retrieve section statement index, critical score value is fixed as 0.80 (ii). Same as (i), but the critical score value is reduced to 0.70 (iii). Preprocessing followed by BM25 technique (iv). Same as (iii), but along with EKB (v). Same as (iii), followed by semantic rule based ranking and retrieval. From the results, it has been inferred that semantic rule based ranking and retrieval gives better performance than the other combinations. Moreover, the
performance is still relatively close to the majority class baseline because most of the conclusion are “entailment” and hence the number of false negatives is less which leads the maximum recall value. But the probability of false positive is relatively high which leads to the precision value of 0.558, which is moderate value. The final result of the leaderboard is given in Table 2 where our team achieved 14th and 19th place in the listed ranks.
The quantitative analysis of the subtasks are, (i). Some statements in the section are indirectly deriving the conclusion and it is resolved by incorporating semantic based rules like negation equivalence, double negation, deductive reasoning and condition based equivalence rules (ii). Adopting TF-IDF
technique retrieves the suitable Clinical Trial Report (CTR) statements efficiently among the CTR section statements (usually each section has 4 to 32 statements) (iii). The RBF-Kernel distance similarity performs linear manipulation to map points to higher dimensional space and hence it is better than other distance similarity metrics. (iv). Most of the hypothesis are entailment (“true”) and hence computing primary and secondary index is feasible.
The error analysis and the solution for the subtasks are, (i). The pairwise distance similarity (Run 4 of subtask 1) measures distance between two points or matrices and hence it degrades the overall performance. Incorporating the RBF-Kernel distance similarity from Run 10 improves the performance of the system (ii). In Run 3 of subtask 1, the non-optimal critical score reduces nearly 30% of the f1-score as compared to Run 1. This is addressed by fixing the suitable critical score in Run 14 (iii). The EKB is used from 10th run and hence suitable critical score value changes and performance also increased. (iv). Run 38 and Run 1 (post competition) of subtask 2 shows that EKB and semantic based rules improved the overall performance of the system.

heading: 6 Conclusion and Future Work
text: In this paper, the Semantic Rule based Clinical Data Analysis (SRCDA) approach is developed for multi-evidence Natural Language Inference (NLI) for clinical trial data task. In SRCDA approach, negation equivalence, double negation, deductive reasoning and condition based equivalence rules were used along with other textual entailment and evidence retrieval techniques. Hence, the F1-score is increased by 17.5% and 23.7% respectively for subtask 1 and subtask 2 respectively. But the overall performance of the system lacks by 18.9% and 14.0% as compared with the leading team in terms of F1-Score.
In the future work, the performance of the model can be improved by incorporating more semantic based rules like transitive and abductive rules, developing an improved External Knowledge Base (EKB) by including more synonyms words from different open sources, and choosing suitable critical score depending upon the contribution of premises towards deriving the conclusion are the ideas (plans) to be executed for improvement. . By incorporating the above discussed ideas, the researchers can develop an improved system which
can be used in other domains like question answering system (chatbot), text summarization, dialogue to text conversion, etc.

heading: Future Work