THiFLY Research at SemEval-2023 Task 7: A Multi-granularity System for CTR-based Textual Entailment and Evidence Retrieval
Year: 2023

Abstract:
The NLI4CT task aims to entail hypotheses based on Clinical Trial Reports (CTRs) and retrieve the corresponding evidence supporting the justification. This task poses a significant challenge, as verifying hypotheses in the NLI4CT task requires the integration of multiple pieces of evidence from one or two CTR(s) and the application of diverse levels of reasoning, including textual and numerical. To address these problems, we present a multi-granularity system for CTR-based textual entailment and evidence retrieval in this paper. Specifically, we construct a Multigranularity Inference Network (MGNet) that exploits sentence-level and token-level encoding to handle both textual entailment and evidence retrieval tasks. Moreover, we enhance the numerical inference capability of the system by leveraging a T5-based model, SciFive, which is pre-trained on the medical corpus. Model ensembling and a joint inference method are further utilized in the system to increase the stability and consistency of inference. The system achieves f1-scores of 0.856 and 0.853 on textual entailment and evidence retrieval tasks, resulting in the best performance on both subtasks. The experimental results corroborate the effectiveness of our proposed method. Our code is publicly available at https:// github.com/THUMLP/NLI4CT.

heading: 1 Introduction
text: In recent years, with the fast development of digital health, there has been a surge in the publication of Clinical Trial Reports (CTRs). Currently, there are over 10,000 CTRs for Breast Cancer alone1. This proliferation of CTRs has enabled the construction of a Natural Language Inference (NLI) system that can aid medical interpretation and evidence retrieval for personalized evidence-based care. The
*: equal contribution 1https://sites.google.com/view/nli4ct/
home
NLI4CT task (Jullien et al., 2023) focuses on constructing an explainable multi-evidence NLI system based on CTRs for Breast Cancer. It consists of two sub-tasks: textual entailment (Task A) and evidence retrieval (Task B). Task A requires the system to determine if the CTRs entail or contradict the given hypotheses. Task B requires the system to find evidence in the CTRs necessary to justify the prediction in Task A. The primary difficulty inherent in NLI4CT is that the process of hypothesis verification frequently involves the integration of several pieces of evidence from the premise. In certain cases, justifying a hypothesis requires a comparison of two separate premise CTRs. Moreover, unlike other clinical NLI benchmarks (e.g., MedNLI), NLI4CT presents a unique feature containing four distinct types of premises - Intervention, Eligibility, Results, and Adverse Events - that align with the four sections typically found in each CTR. Validating hypotheses based on each premise type requires different levels of inference skills (textual, numerical, etc.). Fig.1 presents an example of NLI4CT. The first hypothesis involves
1681
summating the numbers of patients who experienced Diarrhoea in two distinct cohorts to conduct the verification. Alternatively, verifying the second hypothesis requires evaluating treatment modality against the criteria outlined in the Exclusion Criteria.
This paper proposes a multi-granularity inference system for the NLI4CT task to address the aforementioned challenges. An overview of the proposed system is presented in Fig.2. The system comprises a multi-granularity inference network that performs joint semantics encoding of the hypothesis and premise, followed by multigranularity inference through sentence-level and token-level encoding of the encoded sequence to accomplish Task A and Task B. To handle the various levels of inference involved, we utilize the SciFive model (Phan et al., 2021) to enhance the system’s performance on hypotheses requiring numerical inference skills. Our system is presented as an ensemble of models to increase the stability and consistency of inference. A joint inference module is further proposed to refine the inference outcomes of hypotheses by detecting semantic consistency between hypothesis pairs sharing the same premise. Our system achieves micro f1-scores of 0.856 and 0.853 on Task A and Task B, respectively. The main contributions of this work are as follows:
• We present a multi-granularity inference network that performs both textual entailment and evidence retrieval tasks concurrently. We further incorporate the SciFive model to strengthen the system’s numerical inference capability.
• We introduce a joint inference module that improves the inference consistency by exploiting the polarity of hypotheses pairs that share the same premise.
• Our proposed framework achieves state-ofthe-art f1-scores of 0.856 and 0.853 on Task A and B, respectively.

heading: 2 Related Work
text: BiolinkBERT BiolinkBERT (Yasunaga et al., 2022) is a pre-trained model that utilizes hyperlinks within documents. The model incorporates linked documents into the input of pre-training and is pre-trained using masked language modeling and document relation prediction tasks. BiolinkBERT has demonstrated state-of-the-art performance on various medical NLP benchmarks, such as BLURB (Gu et al., 2021) and BioASQ (Nentidis et al., 2020).
SciFive SciFive (Phan et al., 2021) is a T5based model pre-trained on large biomedical corpora, namely PubMed Abstract and PubMed Central. This model has shown the potential of textgenerating models in medical NLP tasks and has achieved state-of-the-art performance on MedNLI (Romanov and Shivade, 2018) and DDI (HerreroZazo et al., 2013).

heading: 3 Task Formulation
text: The Multi-evidence Natural Language Inference for Clinical Trial Data (NLI4CT) task comprises two sub-tasks: textual entailment and evidence
retrieval. Both sub-tasks share a common input comprising a premise P and a hypothesis S. The hypothesis is a single sentence, while the premise comprises several sentences in the CTR(s), i.e., P = [P 1, P 2, · · · , Pm].
The textual entailment sub-task can be viewed as a binary classification problem, with the entailment module fEnt generating a predicted result ŷ ∈ {0, 1}n for all input hypotheses: fEnt(S, P ) = ŷ. In this context, the predicted value ŷ = 1 indicates that the premise entails the hypothesis, while ŷ = 0 denotes that the hypothesis contradicts the premise. On the other hand, the evidence retrieval sub-task can be seen as a binary classification problem targeting each sentence in the premise. The evidence retrieval module fRet produces a predicted result r̂ ∈ {0, 1}m: fRet(S, P ) = r̂, where r̂j = 1 indicates that the jth sentence in the premise supports the justification of S, r̂j = 0 means the sentence is not related to the justification of S.

heading: 4 Models
text: In this section, we introduce models applied in our inference system: Multi-granularity Inference Network (MGNet), SciFive (Phan et al., 2021), and the joint inference network.

heading: 4.1 Multi-granularity Inference Network
text: The multi-granularity inference network aims to handle both the textual entailment and evidence retrieval subtasks using integrated token-level and
sentence-level representations. The network structure is illustrated in Figure 3. This section is organized as follows: Section 4.1.1 presents the joint semantics encoder, which encodes the contextual semantics between the hypothesis and the premise. Section 4.1.2 describes the sentence-level encoder, which learns the sentence-level contextual semantics of the hypothesis and the premise. Section 4.1.3 describes the token-level encoder, which extracts the token-level joint representation between the hypothesis and each sentence in the premise. Finally, Section 4.1.4 introduces the classifier module, which accomplishes Task A and Task B inference based on the learned multi-granularity semantics.

heading: 4.1.1 Joint Semantics Encoder
text: The network initially employs a transformer-based (Vaswani et al., 2017) language model to learn the joint contextual representation of the hypothesis and premise, aiming to incorporate medical domain knowledge into the network. Specifically, we concatenate each hypothesis S with the premise P to form an input token sequence X = [[CLS],S, [SEP],P, [SEP]], where S and P refer to the tokenization of S and P , respectively. Further details regarding data preprocessing can be found at Sec.B in the appendix. The token sequence is subsequently embedded and encoded by the transformer-based language model as follows:
H0 = fEmb(X) (1)
Hl = f lEnc(H l−1) (2)
where H0 denotes the output of the embedding layer, Hl ∈ RN×d (l ∈ {1, 2, · · · , L}) refers to the output representation of the lth transformer encoder layer, N is the sequence length and d the dimension of the representation vector. Here, fEmb and f lEnc refer to the embedding layer and the l th encoding layer, respectively. Finally, the output representation of the final layer, HL, is further processed in the subsequent encoders.

heading: 4.1.2 Sentence-level Encoder
text: The sentence-level encoder conducts contextual encoding of the hypothesis and the premise based on the learned token-level representation. Specifically, the encoder first conducts max-pooling2 to extract
2We have tried other pooling methods (mean-pooling, attention pooling, etc.). However, they fail to achieve higher performance than max-pooling.
sentence-level representation from the token-level representation of each sentence, as follows:
Ĥ s i = Max-pooling({HLj : j ∈ Ii}) (3)
where Ĥ s ∈ R(m+1)×d refers to the pooled sentence-level representation and Ii denotes the subscript set of X that corresponds to the tokens in the ith sentence. For example, Ĥ s 1 corresponds to the hypothesis, while Ĥ s i+1 corresponds to the i th sentence in the premise. After that, the resulting Ĥ s output is fed into an encoder designed to capture contextual semantics. The motivation for this encoder stems from the need to justify some hypotheses using multiple pieces of evidence. To achieve this, we implement the encoder by two alternative networks:
(1) BiLSTM (Hochreiter and Schmidhuber, 1997): the network utilizes two LSTMs to compute the representation of the given sequence from both directions:
→ H s = LSTMsfor(Ĥ s ) (4)
← H s = LSTMsrev(Ĥ s ) (5)
The final representation of each sentence Hs ∈ R(m+1)×d is obtained by concatenating the forward and backward representations and passing them to a linear layer:
Hsi = W1[ → H s i ; ← H s i ] + b1 (6)
where W1 ∈ Rd×2d,b1 ∈ Rd are trainable parameters. Finally, the global representation of the hypothesis-premise pair h̃s ∈ Rd is computed by concatenating the last hidden state of the forward and backward encoding (W2 ∈ Rd×2d,b2 ∈ Rd are also trainable parameters):
h̃s = W2[ → H s m+1; ← H s 1] + b2 (7)
(2) Transformer (Vaswani et al., 2017): a Lslayer transformer encoder integrates reasoningrelated evidence with the sentence-level attention mechanism. In this case, we take the last hidden state as the final representation of each sentence and the hypothesis’s representation as the global representation:
Hs = fsEnc(Ĥ s ) (8)
h̃s = Hsi (9)

heading: 4.1.3 Token-level Encoder
text: The purpose of the token-level encoder is to capture the reasoning-related semantics of each sentence in the premise. Compared to the sentence-level encoder, it provides a fine-grained representation of individual sentences in premise, which has aided in evidence retrieval. To encode the ith sentence in the premise, we concatenate it with the hypothesis and input it into the encoder. Two alternative networks can be used to implement the encoder:
(1) BiLSTM: A BiLSTM network is leveraged to encode the forward and backward representation of the given sequence:
→ H t
i = LSTM t for([H L I1 ,H L Ii+1 ]) (10)
← H t
i = LSTM t rev([H L I1 ,H L Ii+1 ]) (11)
where HLIj ∈ RNj×d refers to the representation of tokens from the jth sentence in X, Nj denotes the length of the jth sentence, [·, ·] denotes the concatenation of matrices on the first dimension, → H t i, ← H t
i ∈ R(N1+Ni+1)×d are the output representation of the LSTMs. Finally, the representation of the ith sentence in the premise is also obtained by concatenating the last hidden state of LSTMs:
Hti = W3[ → H t N1+Ni+1 ; ← H t 1] + b3 (12)
where W3 ∈ Rd×2d and b3 ∈ Rd are trainable parameters as well.
(2) max-pooling: A simpler implementation of the encoder is a max-pooling layer that merges the representation of the sequence:
Hti = Max-pooling(([H L I1 ,H L Ii+1 ])) (13)

heading: 4.1.4 Classifiers
text: We implement the classifiers with simple structures for both Task A and B. For Task A (textual entailment), we use a double-layer MLP that takes the sentence-level global representation h̃s as the input:
pA = Softmax(WA2 σ(W A 1 h̃ s + bA1 ) + b A 2 ) (14)
where WA1 ∈ Rd×d,bA1 ∈ Rd,WA2 ∈ R2×d,bA2 ∈ R2 are MLP’s parameters, σ denotes the GELU activation function (Hendrycks and Gimpel, 2016). The output pA suggests that the given hypothesis has a pA1 probability of being false and a p A 2 probability of being true based on the premise. For Task
B (evidence retrieval), we project the concatenation of sentence-level and token-level representations into a scalar value to determine whether the ith sentence in the premise supports the hypothesis:
pBi = Sigmoid(W B[Hsi ;H t i] + b B) (15)
where WB ∈ R1×2d, bB ∈ R are trainable parameters. pBi measures the probability that the i th sentence in the premise supports the justification of the hypothesis.

heading: 4.2 SciFive
text: The proposed MGNet performs well on Task A and Task B through a unified multi-granularity encoding process. Nevertheless, we found in experiments that MGNet struggles with hypotheses that require numerical inference skills. Motivated, we implement SciFive to enhance the system’s numerical inference ability. Following the original paper, we preprocess the input sequence X in the format of “nli hypothesis: [hypothesis] premise: [premise]". SciFive then computes the probabilities of generating “entailment" (P (“ent”|X)) and “contradiction" P (“con”|X) given the input sequence. The prediction for Task A is then calculated using the following equations:
pA1 = P (“con”|X)
P (“con”|X) + P (“ent”|X) (16)
pA2 = P (“ent”|X)
P (“con”|X) + P (“ent”|X) (17)

heading: 4.3 Joint Inference Network
text: The inference models we implemented have demonstrated satisfactory performance in justifying single hypotheses. However, these models tend to produce identical predictions for hypotheses that share the same premise but contradict each other, leading to inaccurate results. For example, consider the hypotheses “Heart-related adverse events were recorded in both the primary trial and the secondary trial" and “Heart-related adverse events were recorded in the secondary trial, but not the primary trial," which are semantically exclusive. Assigning the same label to both hypotheses is an unreasonable outcome. It is worth noting that such a problem will likely occur in real-world evidencebased care systems, where users may interact with the system multiple times and generate multiple hypotheses based on the same premise.
Motivated by this, we aim to enhance inference consistency and performance by leveraging
the mutual information among hypotheses that share the same premise. To achieve this, we consider a set of hypotheses S1, S2, · · · , Sn that share the same premise and design a joint inference network that determines whether a given pair of hypotheses has the same label for Task A under the given premise. We implement the network with a L-layer transformer encoder. The network initially takes the sequence X̄ = [[CLS],Si, [SEP],Sj , [SEP],P, [SEP]] as input:
H̄L = fJEnc(X̄) (18)
where fJEnc refers to the L-layer transformer encoder. A double-layer MLP is applied to make predictions based on the last hidden state of [CLS] that is considered as the representation vector of the entire sequence:
ci,j = Softmax(WJ2σ(W J 1 H̄ L + bJ1 ) + b J 2 ) (19)
where WJ1 ∈ Rd×d,bJ1 ∈ Rd,WJ2 ∈ R2×d,bJ2 ∈ R2 are trainable parameters and σ represents the Tanh activation function. ci,j1 measures the probability that Si and Sj share the same label, and ci,j2 measures the probability of having different labels. The joint inference module rectifies the predictions of Task A with the prediction results of this network, and we will introduce the details in the next section.

heading: 5 System Details
text: Model Setup We initialized the joint semantics encoder in MGNet and the joint inference network with BiolinkBERT-Large-MNLI-SNLI, a language model proposed by Yasunaga et al. (2022) that is pre-trained on the PubMed dataset and further finetuned on the Multi-Genre Natural Language Inference (MNLI) (Williams et al., 2018) and Stanford Natural Language Inference (SNLI) datasets (Bowman et al., 2015). Following the setting of BiolinkBERT-Large, we set the number of layers L and the dimension of hidden states d to 24 and 1024, respectively. We apply Adam (Kingma and Ba, 2015) and Adafactor optimizers (Shazeer and Stern, 2018) for finetuning MGNet and SciFive, respectively.
To provide a comprehensive description of the inference models used in our system, we first outline the loss functions employed in model training:
(1) Cross-entropy loss for textual entailment:
LA = −(1− y) · log pA1 − y · log pA2 (20)
Here, y ∈ {0, 1} represents the label of Task A. (2) Cross-entropy loss for evidence retrieval:
LB = − 1
m
m∑
i=1
[(1−ri)·log (1− pBi )+ri ·log pBi ]
(21) where ri ∈ {0, 1} is the label of the ith sentence in the premise.
(3) Multitask learning loss: we use a multitask learning loss, which is a combination of the CE loss for Task A and Task B:
Lmul = LA + λLB (22)
Here, λ is a hyperparameter that controls the pace of learning on both tasks.
(4) Contrastive learning loss: we use the contrastive learning loss proposed by Gunel et al. (2020), which is a combination of the CE loss and the supervised contrastive learning (SCL) loss:
LCL = γLA + (1− γ)LSCL (23)
Here, LSCL is the SCL loss that takes the global representation h̃s1, h̃ s 2, · · · , h̃ s N and the labels y1, y2, · · · , yN in the batch as input (N is the batch size). Since our dataset is relatively small, we apply the SCL loss as a regularization technique during training.
The inference models applied in the proposed system are listed in Table.1. To extend the maximum input length of the original BiolinkBERT model from 512 to 1024, we append the positional embeddings of BiolinkBERT with randomly initialized parameters. In Task A, we set the number of layers Ls for the transformer in the sentence-level encoder as one and the learning rates of the joint semantics encoder and other parts as 2e-5 and 1e-4, respectively. The batch size, number of epochs,
and warmup ratio are set to 32, 100, and 0.3, respectively. For SciFive, we set the learning rate, batch size, number of epochs, and warmup steps as 3e-5, 32, 100, and 500, respectively. The hyperparameter values λ = 0.01 and γ = 0.5 are chosen, and the temperature of the supervised contrastive learning (SCL) loss is set to 0.3. In Task B, we set the learning rate, Ls, batch size, number of epochs, and warmup ratio as 5e-6, 2, 1, 50, and 0.05, respectively.
In the joint inference network, the learning rate, batch size, number of epochs, and warmup ratio are set to 2e-5, 32, 100, and 0.3, respectively. Positive samples, i.e., hypothesis pairs with the same label, are created using back translation, while negative samples are obtained by identifying hypothesis pairs with different labels that appear in the datasets. Further training details of this network are presented in Section C in the appendix, owing to space limitations.
Soft Ensembling We apply a cross-validationbased soft ensembling method to summarize the inference results from different models. Specifically, we perform 10-fold validation for each model in Table.1 using all data in the train and dev sets. For Task A, we save the checkpoint that achieves the best f1-score on the corresponding dev set for each fold, while for Task B, we save two of the best checkpoints for each fold. Then, we average the predictions of 4×10 = 40 models trained by crossvalidation for Task A and average the predictions of 3 × 10 × 2 + 3 = 63 models trained by crossvalidation (two checkpoints for each fold) and holdout method (one checkpoint for each model) for soft ensembling. The ensemble predictions for the two tasks are denoted as p̃A and p̃B , respectively.
Joint Inference As mentioned above, we utilize a joint inference module to rectify the ensembling results p̃A before performing Task A. This module corrects the prediction of hypotheses that share the same premise using the following equation:
p̂A,i = 1
n
n∑
j=1
(Iij p̃A,j+(1−Iij)(1− p̃A,j)) (24)
Here, p̃A,i represents the prediction of the ith hypothesis after ensembling, while ci,j1 denotes the prediction results of the joint inference network discussed in Sec.4.3. The function Iij = I[ci,j1 > 0.5] employs an indicative function I[·], which is equal to 1 if the condition in brackets holds, and 0 otherwise. We set Iii = 1 since two identical hypotheses must share the same labels. Notably, when n = 2 and the two hypotheses are predicted to be contradicting each other (c1,21 < 0.5), the hypothesis with a higher confidence level in its prediction retains its original prediction, while the hypothesis with a lower confidence level modifies its prediction. Therefore, the joint inference module improves the system’s prediction consistency.
System Output We use a threshold-based decision strategy in both tasks to obtain the ultimate predictions. Specifically, for Task A, hypotheses with p̂A2 > ηA are predicted as “entailment", while the remaining hypotheses are predicted as “contradic-
tion." For Task B, the system determines whether the ith sentence in the premise supports the inference by checking if p̃Bi > ηB . In our implementation, we set the threshold values to ηA = 0.57 and ηB = 0.53.

heading: 6 Evaluation

heading: 6.1 Data, Metrics, and Baselines
text: We conduct all the evaluations on the NLI4CT dataset. The train, dev, and test sets contain 1,700, 200, and 500 hypotheses. The hypotheses can be divided by the number of CTRs involved: (1) Single, related to only one CTR; (2) Comparison, requiring the comparison between two CTRs for justification. The premise is categorized into four sections that correspond to the CTR sections: (1) Intervention; (2) Eligibility; (3) Results; (4) Adverse Events. Additional dataset details are available in Sec.A in the appendix. Recall, precision, and f1-score are chosen as the evaluation metrics for both Task A and Task B.
We compare our system with several baseline models (BiolinkBERT-Large, SciFive) that achieve SOTA performance on medical NLI tasks. For the BiolinkBERT-Large model, we feed the concatenated hypothesis and premise sequence into the language model and add an MLP that takes the representation of [CLS] as input for classification. Note that the performance of baseline models is obtained by ensembling 10 models through crossvalidation to make the results comparable. We did not train baseline models for Task B because it is hard to find an existing implementation that can directly be applied to Task B.

heading: 6.2 Overall Performance
text: The results of the proposed system on Task A and B are presented in Table 2 and 3, respectively. Our
system performs best on both tasks with f1-scores of 0.856 and 0.853 for Task A and Task B, respectively. Compared with the baseline models, our system shows a significant improvement of approximately 3% and 5% on the f1-score with and without the joint inference module, respectively, demonstrating the effectiveness of our approach. Note that the proposed multi-granularity encoding method greatly improves the performance of the original BiolinkBERT-Large model on Task A (- SciFive vs. BiolinkBERT-Large) by around 6% in f1-score (0.753 vs. 0.693, 0.844 vs. 0.768).

heading: 6.3 Ablation Study
text: We performed ablation experiments to investigate the effect of the proposed joint inference method and the contribution of each model in the soft ensembling. The results are presented in Table 2 and 3. The results show that the joint inference module leads to a significant improvement of 8% points on Task A, suggesting that the system is more likely to make the correct prediction when considering the mutual information between hypotheses, i.e., whether the hypotheses share the same label.
The experimental results suggest that the MGNet, which employs both sentence-level and token-level encoders implemented by BiLSTM (M512-Bi-Bi/M-512-Bi-Bi-mul), plays a crucial role in both Task A and B, contributing to 1.8% and 1.3% improvements in f1-score, respectively. Furthermore, the individual inference case results reveal that SciFive provides the most significant improvement in the system’s performance, with a 2.3% increase in the f1-score.
To further investigate the impact of multigranularity encoding in MGNet, we conduct an additional ablation experiment on Task B by removing either the sentence-level or token-level encoder (Task A only relies on the sentence-level encoder). The results in Table 4 show that both sentence-level and token-level encoders enhance the system’s performance, with the sentence-level
encoder contributing more to the dev set (4% improvement) and the token-level encoder contributing more to the test set (1.3% improvement). The results indicate the effectiveness of the proposed multi-granularity encoding.

heading: 6.4 Performance on Different Sections
text: We further examine the behavior of each model that contributes to the ensembling by evaluating the performance of our system on different sections of CTR. The analysis results are presented in Table 5. It can be observed that MGNets demonstrate superior performance on hypotheses involving the Intervention, Eligibility, and Results sections. In contrast, SciFive achieves a higher f1score on hypotheses related to the Adverse Events section. Note that hypotheses concerning the Adverse Events section generally require more numerical calculations than those related to other sections. These results demonstrate that SciFive enhances the system’s performance with its numerical inference ability.

heading: 7 Conclusion
text: We propose a Multi-granularity system for CTRbased textual entailment and evidence retrieval. The system comprises an ensembling of inference networks and a joint inference module to enhance prediction consistency. A Multi-granularity Inference Network (MGNet) is proposed to handle the two subtasks within a unified network. Another text-generation-based network SciFive is also leveraged to enhance the numerical inference of the system. We use model ensembling to integrate different models’ output and enhance the inference stability, while a joint inference method is further proposed to increase the consistency of inference. Our system achieves the best performance on both subtasks. The ablation experiments and analysis indicate the effectiveness of different parts of the system. In the future, we will try other large language
models and new inference methods (e.g, in-context learning) to inject more medical knowledge into the system and further improve the performance on this task.

heading: A Details of Data
text: Table 6 shows the basic statistics of NLI4CT. Avg Hypo denotes the average number of words in hypotheses, Avg CTR Sents refers to the average number of sentences in CTRs, and Avg CTR Words is the average number of words in CTRs.
Fig.4 depicts the distribution of the input sequences (concatenation of the hypothesis with premise) regarding the number of tokens in sequences. Most input sequences have less than 500 tokens, while there also exists input sequence longer than 1750 tokens.
As for the CTR, each CTR contains a total of four sections:3
• Intervention: Information concerning the type, dosage, frequency, and duration of treatments being studied.
• Eligibility: A set of conditions for patients to be allowed to take part in the clinical trial
• Results: Number of participants in the trial, outcome measures, units, and the results.
• Adverse Events: These are signs and symptoms observed in patients during the clinical trial.
3The definition of each section is from https://sites. google.com/view/nli4ct/home.
It is worth to note that each hypothesis only involves a single section in CTR. For hypotheses that involve multiple CTRs, one is required to compare the same section of different CTRs.

heading: B Data Preprocessing
text: We process the evidence CTRs differently for Task A and B. For hypotheses that require single CTR, we simply concatenate all the sentences in the CTR to form the premise sequence for Task A and B. For hypotheses that require comparison between two CTRs, we concatenate the first and second CTR in a single sequence for Task A with the following format: “primary trial: [CTR 1]. secondary trial: [CTR 2]." Therefore, the joint semantics between two CTRs can be captured by the language model. Nevertheless, we find that encoding each CTR with the hypothesis achieves higher performance for Task B. Therefore, we conduct evidence retrieval on each CTR individually.

heading: C Details of Training Joint Inference Network
text: It is hard to train the joint inference network on NLI4CT, because most of the premise corresponds to a pair of hypotheses that contradicts with each other and we can’t find positive samples (hypotheses that share the same label). To address this problem, we use the back-translate method to generate positive samples. For each hypothesis S in the train and dev set, we translate it to Chinese and back translate to English to get a new hypothesis S′. Then S and S′ share the same label for Task A. For a pair of conflicting hypotheses S1, S2, we generate four input sequences for the joint inference network:
1. [[CLS],S1, [SEP],S′1, [SEP],P, [SEP]]
2. [[CLS],S2, [SEP],S′2, [SEP],P, [SEP]]
3. [[CLS],S1, [SEP],S2, [SEP],P, [SEP]]
4. [[CLS],S2, [SEP],S1, [SEP],P, [SEP]]
After the data generation, we train the joint inference network on the train set and choose the best model on the dev set. We set the learning rate, warmup ratio, and batch size as 2e-5, 0.1 and 32 respectively.

heading: Future Work
text: In the future, we will try other large language
models and new inference methods (e. g, in-context learning) to inject more medical knowledge into the system and further improve the performance on this task