NLI4CT: Multi-Evidence Natural Language Inference for Clinical Trial Reports
Authors: Mael Jullien
Marco Valentino
Hannah Ruth Frost
Paul O'Regan
Dónal Landers
Andre Freitas
Venue: EMNLP 2023 Main (2023)
Keywords: NLI
Clinical Trial
Textual entailment
Evidence retrieval
NLP
TL;DR: Baselines and description of a novel NLP dataset with two tasks, textual entailment, and evidence retrieval, based on cancer clinical trials

Abstract: How can we interpret and retrieve medical evidence to support clinical decisions? Clinical trial reports (CTR) amassed over the years contain indispensable information for the development of personalized medicine. However, it is practically infeasible to manually inspect over 400,000+ clinical trial reports in order to find the best evidence for experimental treatments. Natural Language Inference (NLI) offers a potential solution to this problem, by allowing the scalable computation of textual entailment. However, existing NLI models perform poorly on biomedical corpora, and previously published datasets fail to capture the full complexity of inference over CTRs.

In this work, we present a novel resource to advance research on NLI for reasoning on CTRs. The resource includes two main tasks. Firstly, to determine the inference relation between a natural language statement, and a CTR. Secondly, to retrieve supporting facts to justify the predicted relation. We provide NLI4CT, a corpus of 2400 statements and CTRs, annotated for these tasks. Baselines on this corpus expose the limitations of existing NLI approaches, with 6 state-of-the-art NLI models achieving a maximum F1 score of 0.627. To the best of our knowledge, we are the first to design a task that covers the interpretation of full CTRs. To encourage further work on this challenging dataset, we make the corpus, competition leaderboard, and website, available on CodaLab, and code to replicate the baseline experiments on GitHub.

NLI4CT: Multi-Evidence Natural Language Inference for Clinical Trial Reports
Maël Jullien', Marco Valentino', Hannah Frost!, Paul O' O'Regan', Donal Landers,, André Freitas1.2,3
Department of Computer Science, University of Manchester, United Kingdom Digital Experimental Cancer Medicine Team, Cancer Research UK Manchester Institute
'Idiap Research Institute, Switzerland
Abstract
NLACT Task
Hop Ressoning
Patueru living Frauctee
How can we interpret and retrieve medical ev- idence to support clinical decisions? Clinical trial reports (CTR) amassed over the years con- tain indispensable information for the devel- opment of personalized medicine. However, it is practically infeasible to manually inspect over 400,000+ clinical trial reports in order to find the best evidence for experimental treat- ments. Natural Language Inference (NIJ)) of- fers a potential solution to this problem, by allowing the scalable computation of textual entailment. However, existing NLJ models per- form poorly on biomedical corpora, and previ- ously published datasets fail to capture the full complexity of inference over CTRs.
B1802 hepas cansic body wullfe
oxtemice 40s
tyydashersheadde it the lus yoar
Hightiy Orharie
cnevessie (iara MMorkat 1x 12:22
tackukea Crieetic
Certessnoema Reswetir
poodie buiilegioddy cuntrrrd teaatre af Ienars
Taaciaa Sin Frietire- di Capleno
Boll sugga 1001hs
Blsnedical Tnfervase
Patecn
Chenuchinge taanaae withii
Callicnls sarkara
Aod protiics
Racaasia Criteria:
Figure 1: We propose two tasks for reasoning on clinical trial data expressed in natural language. Firstly, to pre- dict the entailment of Statement and CTR premise, and secondly, to extract evidence to support the label.
In this work, we present novel resource to ad- vance research on IL. for reasoning on CTRs. The resource includes two main tasks. Firstly, to determine the inference relation between a natural language statement, and a CTR. Sec- ondly, to retrieve supporting facts to justify the predicted relation. We provide NLI4CT, a corpus of 2400 statements and CTRs, anno- tated for these tasks. Baselines on this cor- pus expose the limitations existing NI.I ap- proaches, with 6 state-of-the-art NLI models achieving a maximum Fl score of 0.627. To the best of our knowledge, we are the first to design a task that covers the interpretation of full CTRs. To encourage further work on this challenging dataset, we make the corpus, com- petition leaderboard, and website, available on CodaLab, and code to replicate the baseline experiments GitHub!.
of a particular trial. Clinical practitioners use the in- formation in these CTRs to design personalized and targeted interventions, matching therapeutic agents to each patient's tumor biomarkers. However, there are over 400,000 CTRs available, being continually published at an ever-increasing rate (Bastian et al., 2010). Consequently, it has become unfeasible to manually carry out comprehensive evaluations of all the relevant literature when designing treatment protocols Youno et al., 2020). Natural Lan- guage Inference (NLI) offers potential solution for the large-scale interpretation and retrieval of medical evidence, connecting the latest evidence support personalized care (Sutton et al., 2020).
1 Introduction
In this paper, we propose two different tasks, on breast cancer CTRs. Firstly, to determine the infer- ence relation between natural language statement and CTR, as shown in Figure 1. Secondly, retrieve supporting facts from the CTR(s) to justify the predicted relation. With respect to healthcare, these inference tasks are a move towards automatic claim verification and evidence extraction, over
Clinical trials are research studies performed to test the efficacy and safety of novel treatments, and they are indispensable for the progression of ex- perimental medicine (Avis et al., 2006). CTRs are documents that detail the methodology and results
lemail: mael. jull www.emmnchetttrrac....

---

biomedical corpora.
synthesis, and inference of published literature support evidence-based experimental medicine.
These proposed tasks represent several chal- lenges for existing NLI systems. The inference task requires substantial amounts of quantitative reason- ing and numerical operations, as seen Figure 1, on which many current state-of-the-art NLI models perform poorly (Ravichander et al., 2019; Galashov et al., 2019). Additionally, much of the inference for these tasks revolves around domain-specific terminology (see Figure 1). Many lfae-of-theearr art NLI models fail to effectively surmount the word distribution shift from general domain corpora to biomedical corpora through transfer learning (Lee et al., 2019). This phenomenon is well-represented in the proposed dataset. This task differentiates itself from other biomedical NLI tasks by consid- ering the full scope of CTRs, not limited to results and interventions, instead including eligibility and Adverse events (AE) as well (De et al., 2020). The proposed task also provides significant vari- ance in the types of inference required to solve each instance, with minimal, any, repetition in inference chains. The contributions of this paper are as follows:
Related Work
There are a multitude of expert-annotated resources for clinical NLP, examples are shown in Table 1. The TREC 2021 Clinical Track (Soboroff, 2021) is large-scale information retrieval task on CTR data, with an emphasis eligibilitty. Evidence Inference 2.0 (DeYoung et al.. 2020) presents a Prestion-Answering (QA) task and a span selec- tion task on CTR data, specifically with the CTR re- sults. The MEDNLI (Romanov and Shivade, 2018) dataset contains an entailment task, which utilizes the medical history notes of patients as premises. These datasets are predominantly designed to test for biomedical language understanding and rea- soning. Whereas NLI4CT, additionally requires complex numerical inference and common-sense reasoning to solve and designed avoid repeti- tive inference patterns. Currently, neural architec- tures achieve the best results on biomedical NLI datasets (Gu et al., 2021; ; De Young et al., 2020).
Neural models continue to perform poorly quantitative reasoning and numerical operations within NLI (Ravichander et al., 2019; Galashov et al., 2019). Prior works such as Lee et al. (2020), Shin et al. (2020), and Gu et al. (2021) experi- ment with biomedical pre-training strategies. Kir- itchenko et al. (2010) presents ExaCT, an automatic information extraction system for clinical trial vari- ables such as drug dosage, sample size, and pri- mary outcomes from CTRS. While there are many proficient evidence retrieval models, there are no existing systems that are able to effectively carry out biomedical and numerical NLI simultaneously. In related vein, Hao et al. (2014) introduced automated approach to cluster clinical trials based on similar eligibility features. Using data from ClinicalTrials, features from the eligibility criteria of trials and em- ployed the Jaccard similarity coefficient for cluster- ing. Their method, which generated 8,806 center- based clusters, offers potential utility in understand- ing knowledge reuse patterns in clinical trial de- signs and enhancing recruitment strategies. Liu et al. (2021) presented the Clinical Trial Knowl- edge Base (CTKB), a comprehensive database that houses discrete clinical trial eligibility criteria. De- veloped using advanced NLP techniques, CTKB transforms free-text criteria from ClinicalTrials.ge.
1. The definition of a novel benchmark (NLI4CT), including two main tasks requiring reasoning over CTRs that incorporate several of the fundamental challenges currently faced by modern NLI systems.
2. The release of a newly publicly-available cor- pus containing 2,400 expert-annen -aport-annotated entail- ment relations, with associated CTRs from ClinicalTrials. govr, labels, and extracted evi- dence lists.
3. An extensive empirical evaluation of state-of- the-art NLI models demonstrating current lim- itations and challenges involved in the pro- posed tasks. Specifically, we test represen- tative NLI models on our dataset and report maximum Fl score of 0.644. We attribute these results to generalization problems in- duced by the shift in distribution required for inference.
they extracted semantic
To the best of our knowledge, we are the first to design a task that covers the full scope of CTRs, combining together the complexity of biomedical and numerical NLI. Solving these tasks would be a significant advancement toward efficient retrieval,
Phtres://ciinilclttiiaaa

---

Dataset NLJ4CT (2023)
Task
Metrics
Dataset size
Labels
Predict the entailment of statement and one Macro FI of sections of clinical trial and select
2400 instances
Entailment
Contradiction
evidence to support the label.
Evidence Inference 2.0 (DeYoung et al,, 2020)
Given treatment A, comparator outcome, and a CTR, predict the relationship Evidence token AUC
Macro Fl
10,000+ QA pairs
significantly
increased
between A and B relative to the outcome.
significantly
Then identify spans of evidence in the CTR.
decreased
MEDNLI (Romanov and Shivade, 2018)
Given an annotated sentence, and sentence Accuracy from patient medical records determine the
14,000+ sentence pairs
Entailment
neutral
entailment relation.
Contradiction
TRBC clinical track (Roborts et al, 2021)
Given synthetic patient descriptions/clinial notes retrieve relevant CTRs for which this Precision patient would be eligible.
NDCG
75 synthetic patient cases 350K clinical trials
Eligible
Excludes
Not Relevant
Table Description of the main Clinical NLP datasets present in the literature.
into structured concepts and attributes, facilitating improved querying and analysis. This knowledge base holds promise in enhancing clinical trial co- hort definitions, assessing population representa- tiveness, and bridging data gaps in electronic health records for clinical trial recruitment. In a comple- mentary effort to streamline clinical trial recruit- ment Miotto and Weng (2015), introduced case- based reasoning approach using electronic health records (EHRs). By aggregating the EHR data of a minimal sample of trial participants, they created "target patient" profile for each trial. This profile eligible
corpus, comprising 1200 texts about clinical trials, is annotated with entities from the Unified Medical Language System, covering areas such anatomy, pharmacological substances, pathologies, and di- agnostic or therapeutic procedures. Their work emphasizes the potential of natural language pro- cessing in bridging the gap between vast medical literature and the need for healthcare profession- als to access relevant, evidence-base information efficiently.
Clinical Trial Reports
then used to efficiently identify patients based on their relevance to the "target pa- tient." Their method, evaluated on 13 diversified clinical trials at Columbia University, demonstrated high precision in identifying eligible patients, em- phasizing the capability of EHR-driven solutions in streamlining clinical trial recruitment.
Clinical trials are research studies on human vol- unteers used to evaluate an intervention (De et al., 2020). An intervention may be medical, surgical, or a behavioral protocol. These interven- tions are often compared with a standard treatment, placebo, or control protocol. The investigators use various outcome measurements from the volunteers to ascertain the effectiveness of the intervention, as well as the likelihood and severity of Alis.
As the biomedical domain continues to gener- ate vast amounts of textual data, the importance of efficient concept recognition grows. Kors et al. (2015) introduced the Mantra GSC, multilingual gold-standard corpus designed for biomedical con- cept recognition across five languages: English, French, German, Spanish, and Dutch. This corpus, based on parallel biomedical texts from sources like Medline and drug labels, provides annotations grounded in a subset of the Unified Medical Lan- guage System. Such endeavors underscore the global push towards more multilingual biomedical information extraction tools, catering to the diverse linguistic landscape of the biomedical community. In the pursuit of enhancing access evidence- al. (2021)
For NLI4CT we retrieved 1000 publicly avail- able Breast cancer CTRs, with published results from ClinicalTrials. on the 22nd of December 2021. This data is developed by the U.S. National Library of Medicine and covered by the HIPAA Privacy Rule, protecting personally identifiable in- formation (additional ethical considerations can be found in Section 11). We separate the CTRs into 4
sections:
Eligibility criteria: A set of conditions for patients to be allowed to take part in the clinical trial.
Intervention: The type, dosage, frequency, and duration of treatments being studied. Results: Number of participants in the trial, out- come measures, units, and the results.
based medicine, Campillos-Llanos introduced the Clinical Trials for Evidence-Based Medicine in Spanish (CT-EBM-SP) corpus. This
Adverse Events: These are signs and symptoms

---

Eligibility Criteria
Inclusion:
5 Annotation
HER2 + diagnosis
18 years of age or older
5.1 Statement annotation
Exclusion:
Presence central nervous
A group of 4 domain experts, including clinical trial organizers from the Manchester Cancer Insti- tute and the Digital Experimental Cancer Medicine Team (DECMT), took part in the annotation task, with one expert performing the majority of the an- notations. We supplied annotators with an annota- tion guide, displayed in the appendix, which was refined following the annotation of 100 pilot ex- amples by the principal expert annotator to opti- mize the overall annotation process. Annotators were provided with a CTR section prompt, and two CTRs, a primary and secondary, as seen in step (B) of Figure 2. The annotation task was first generate an entailment statement, a short statement that makes an objectively true claim about the con- tents of the prompted section of the trial(s). In step (C) of Figure 2 annotators could choose to write statement about the contents of the primary trial or compare both the primary and secondary tri- als. The objective was to generate non-trivial state- ments, encouraging solve the inference by read- ing and understanding multiple rows of the CTR section. Non-trivial statements typically included summarisation, comparisons, negation, relations, inclusion, superlatives, aggregations, or rephrasing. However, the sentences were not limited to these types, and any statement involving understanding accepted.
system or brain metastases.
Intervention
Gemeitabine 1500 mg/m2 body surface (BSA) intravenously (IV) 30 minutes (+/- minutes) on days and
of each eycle.
Results
Variable: Progression Free Survival
Number of Participants: 29 Median (months): 10.4 (5.6 to 15.2)
Adverse
Total: 3/29
Events
Neutropenic Fever 29 (3.45%)
Peripheral Neuropathy 1 29 (3.45%)
Scizure/Syncope 29 (3.45%)
Table 2: Sample excerpts from each section in CTR.
Data provided by ClinicalTrials.go
observed in patients during the clinical trial. See examples of the types of information con- tained in these sections in Table 2.
4 Task Definition
We define two different tasks on the NLI4CT dataset, Task 1, a textual entailment task, and Task 2, an evidence selection task. Each instance in NLI4CT contains a CTR premise and a statement. The CTR premise comprises one of the four sec- tions of CTR, with both a mean token length and a standard deviation of 230. The statements are sentences ranging from 10-35 tokens (see example in Figure 1). On average there are 7.74 pieces of relevant evidence that need to be selected out of a total of 21.67 facts per CTR premise. There are two types of instances NLI4CT, in single type instances the CTR premise is the primary CTR, and the statement makes a claim about the information in this CTR. Whereas in the comparison type the premise contains primary and secondary trial, and the statement will make claim comparing and contrasting the two trials. To summarize:
and reasoning
Each CTR is divided into sections (Table 1), and each of these sections represents a set of facts. And for all annotated statements we select a subset of these facts that support the label assigned to the statement, see Figure 1 and (D) in Figure 2. This collection of facts is designated as evidence. There is always at least one piece of evidence relevant to the inference for any given statement. In cases where negation is used, e.g. clinical trial A does not have placebo arm annotators are asked provide the full CTR section as evidence, as we believe this type of retrieval is more reflective human inference patterns.
Task Requires determining the entailment rela- tion between the CTR premise and the statement, therefore the output is either an entailment or con- tradiction label, as shown in Figure 1.
Finally, we employ negative rewriting strategy (Chen et al., 2019), using the previously generated entailment statement to write an objectively wrong contradiction statement, as shown in Step (E) of Figure 2. Annotators are encouraged to modify the words, phrases, or sentence structures while retaining the sentence style/length of the original statement. This done to minimize the occurrence
Requires outputting a set of supporting facts, extracted from the CTR premise, necessary to justify the label predicted in Task 1.
Task 2

---

Library
Section ID: Eigiblly
Statement: Natolopically stage breast adenocarccooee that eiher ER-pceiiive. PRiposlivve HEREheu negalive are algiblo for both the seconcary thal and primary tris.
ClinicalTrials.gon
Primary NCT01156967
Secondary CTR: NCT01129622
(A) Data collection from cliiccatriates.gvv
(B) Prampt generation
(C) Annotation of entailment statement
Primary CTR evidence:
Contradiction Statement: Patients with histolopisall contimed pocatve, postive secondery friel, but not the promery Viel.
Andlusion Gritera:
Aduls with histlogiccaly cytologicaty contimedd breast adenocarchoms
are olgibla
Secondary evidence:
Entena:
Pallents must have Nitollgically confoneed treest cencer posilive andir postive, and HERI/nau negative and have disease that matastalic (stago IV)
[5] Negative wrrting of contradiction statements selection of contradicting
(D) Selection of supporting evidence
Figure 2: Overview of the NLI4CT annotation process.
of stylistic or linguistic patterns pertaining to ei- ther entailment or contradictory statements. We then collect evidence that contradicts the claims made the contradiction statement. An example annotation form is available in the appendix.
et al., 2021). Future systems must overcome this challenge perform biomedical NLI.
Synonyms and Aliases. These are often em- ployed for drugs and gene names. Statements in NLI4CT regularly refer to treatments and diag- noses using different aliases than are present the CTR premise, e.g. the medication Trastuzumab has different brand names, and the gene ERBB2 has over 20 aliases.
For quality assurance, 3 volunteers re-classified 100 randomly sampled test instances, recording average accuracy of 85% with respect gold labels and Cohen's kappa of 0.83, indicating substantial inter-annotator agreement.
Taxonomic Relations. Concepts such as dis- eases, treatments, and diagnostic tests can be clas- sified and structured in taxonomic hierarchy. As shown in Figure 1, to achieve the necessary infer- ence systems must understand that chemotherapy is hypernym super-set of Cyclophosphamide treatments, similar Cancer being hypernym of Carcinoma.
5.2 Resulting Dataset
The NLI4CT dataset consists of 2,400 annotated statements with accompanying labels, CTRs, and evidence. Split into 1700 training, 500 test, and 200 development instances. The two labels and CTR sections prompts are equally distributed across the dataset and its splits. 60% of the instances are the single type, with the remaining 40% the compari- type. The total annotation effort for NL.4CT accumulated to approximately 260 person-hours.
Domain Knowledge. NLI4CT statements regu- larly test for domain expert knowledge. This is done by describing characteristics or conditions which pertain to certain biomedical grading sys- tems or diagnoses. For example, if a patient is confined to bed or chair for more than 50% of waking hours, and the inclusion criteria require WHO score of 2, the model must be able to infer that the patient will not be eligible. Or if patient has positive FISH test, and the inclusion criteria require the patient to have a Triple Negative Breast Cancer diagnosis, they would also be ineligible.
6 Reasoning Challenges
6.1 Biomedical Reasoning
Acronyms. In CTRs acronyms are used for con- cepts such as test names, administration times and orders, treatment types, and disease names. Acronyms are significantly more prevalent in clin- ical texts than in general-domale texts and con- sistently disrupt NL. performance in this field (Grossman Liu et al., 2021; Shickel al., 2017; Jiang et al., 201 1: Moon al., 2015; Jimeno- Yepes al., 2011; Pesaranghader et al., 2019; Jin al.. 2019; Wu al., 2015). Over 100,000 healthcare acronyms and abbreviations have been identified with 170,00 corresponding senses (Grossman Liu
6.2 Commonsense Reasoning
NLI4CT requires common sense reasoning, partic- ularly co-reference resolution and general world knowledge, which remain challenging for existing NLI models (Emami et al., 2018; Trichelair et al.,

---

Turbo because it offers a more cost-effective so- lution than GPT-4 and consistently demonstrates superior performance compared to LLaMa. Ad- ditionally, we experiment with the bag-of-words ranking function Okapi BM25 (Fang et al., 2004; Robertson et al., 1995).
Statement: 'Most patients in cohort sulfered cardiac event,
"Mos!" > 50% > 1/2
Aggregation:
Section:
Adverse events
Inequality:
Adverse Events
Total: 36/33
4/93 1/2
Leukopenia 3/93
7.2 Biomedical Pretraining
Arrhythmia 1/83
Atrial fibrillation 1/93
Prediction:
Transformer-bused models leverage pre-trained contextualized word embeddings to model infer- ence, and optimizing pre-training strategies drastically improve downstream performance (Gu- rurangan et al., 2020). Previous works have repeat- edly shown that pre-training general-domain lan- guage models on biomedical corpora significantly improves performance on bio-medical tasks (Shin et al., 2020; Beltagy et al., 2019; Lee et al.. 2020; Li et al., 2016). Therefore, we also include BioBERT (Lee et al., 2020), and BioMegatron (Shin et al., 2020), versions of BERT-base and Megatron-Im that are pre-trained on biomedical corpora, in our baselines assess the effects of pre-training on per- formance on our dataset. We selected these models as emblematic of the domain-specifie model re- search community, deferring detailed evaluation of such models for future studies.
Pericardial effusion 1/93
Tachycarcia 1/93
Contradlction
Figure 3: Schematic a numerical inference chain to solve NLI4CT instance.
2018). Co-referencing resolution is often necessary associate dosages, frequencies, and routes of ad- ministration to specific drugs in the intervention. General world knowledge most often required for claims about the eligibility section, e.g. a child is a person < 18 years old, and women over the age of 60 are not of childbcaring potential.
6.3 Numerical Reasoning
The type of numerical reasoning required solve instances most closely resembles Math Word Prob- lems (MWP) (Huang et al., 2016; Patel et al., 2021). Prior works have shown that even elemen- tary MWPs remain unsolved for current NLP mod- els (Patel et ul., 2021). NLI4CT requires mod- els to compare dosages, frequencies, and frac- tions/percentages often necessitating unit conver- sions. Consider the example instance in Figure 3. Here the model will need to aggregate the num- ber occurrences of every different cardiac event recorded in the adverse events of cohort 1. Then it must construct and evaluate the inequality. This combination of biomedical knowledge and numer- ical reasoning is expected to be significant chal- lenge for existing models.
7.3 Experimental Details
For the transformer-based model basclines we to- kenize the statements and the section(s) of the CTR(s) as indicated by the instance type and sec- tion prompt. If the instance type is comparison we concatenate the primary and secondary CTR section data together. Then we pass the tokenized statement and section information to the model. It should be noted that for a significant portion of the instances, particularly of the comparison type, the input exceeds the maximum input length of 512 tokens. We employ binary sequence clas- sification/regressionn head for all baseline models, which predicts either Entailement or Contradic- tion. The models were trained on the training set for 10 epochs, and each model checkpoint was archived. We additionally test zero-shot GPT 3.5 turbo and provide the prompt in the appendix. Fi- nally, in line with previous work explainable NLI (Valentino al., 2022a,b; Jansen and Ustalov, 2019; Thayaparan al., 2021) adopt BM25 to derive sparse vectorial sentence representations. and subsequently, employ cosine similarity as a bounded scoring function then apply threshold on the cosine similarity between statements and
7 Experiments
7.1 Baseline Model Architectures
We establish a baseline performance on NLI4CT, to assess the strengths and limitations of existing NLI models. We test range of wansformer-based architectures: BERT-base (Devlin et al., 2018),
RoBERTa-base (Liu al., 2019), GPT-2 (Rad-
ford et al., 2019), T5-base (Raffel et al., 2020),
Megatron-Im (Shocybi et al., 2019), and GPT 3.5
turbo (Brown et al., 2020). We chose GPT-3.5

---

Model
Precision
Recull
Accurucy
BioBERT, and BioMegatron both achieve over 0.95 accuracy and Fl score on the training set after cantly outperforms its general-domain counterpart on the training set, achieving a maximum Fl score of 0.96, in contrast to 0.65 by BERT-base. Biomed- ical pre-training clearly impacts the models' ability to encode relevant information, limited the train- ing set. On average, model complexity does not have a consistent impact on performance. Neither the over-fitting biomedical models nor the models which failed to learn on the training set exhibited any positive responses to changes in the learning rate. The aggregation of these results indicates that the NLI4CT entailment task represents significant challenge for existing NLI models.
BM25
0.627
0.490
0.872
0.482
BERT-basc
0.528
0.555
0.504
0.550
10 epochs. As shown in Table 3, this does not trans- late to an improvement in performance on the test, indicating drastic over-fitting. BioBERT signifi-
RoBERTa-base
0.628
0.654
0.604
0.642
BioBERT
0.644
0.633
0.656
0.638
BioMegatron
0.644
0.585
0.716
0.604
GPT2
0.603
0.500
0.760
0,500
T5-base
0.602
0.504
0.748
0.506
GPT 3.5 turbo
0.604
0.630
0.580
0.620
Table Results from the NLI4CT Task baselines on the test set.
premises to perform binary classification (i.e., pre- dicting entailment if the cosine similarity is above the threshold, contradiction otherwise), we test threshold values from 0.1 0.9. We then record the results of these models on the set. Code to reproduce the experiments is available on GitHub.
9.2 Evidence-Only Baseline
We generated an additional of basclines, using only the gold evidence as the premise. This reduces the likelihood of the models focusing on irrelevant or adversarial information and removes the need to identify evidence spans within the CTR section. This did not result in any significant differences in performance, with BioBERT achieving the high- est Fl score of 0.667. This implies that even after training for 10 epochs none of the models effec- tively draw inferences from the relevant evidence. The results for this baseline are available in the appendix.
8 Evaluation Metries
Task 1 is a binary classification task, the statement is either labeled as entailment or contradiction. We evaluate performance task 1 by computing the Precision, Recall, and Macro Fl-score of the pre- dicted labels against the annotated gold labels (En- (ontennt/ootraaiitioonn... For Task the output a subset of the facts within the CTR premise. We can frame the evidence selection problem as a rank- ing problem (Jansen et al., 2021). Models assign a score to each fact in a CTR premise, which can be used to rank the facts. In this framework, solving the task requires the model rank all the facts in the gold evidence higher than the irrelevant facts. We evaluate performance on this task using Mean Average Precision mAP @ K (Liu, 2011).
9.3 Statistical Artefacts
We test BERT-base, using only the statements, removing any access to the CTRs. BERT-base achieved maximum F1 score of 0.626 and an accuracy of 0.606 on the test set, significantly higher F1 score than the BERT-base Task base- line (0.528). Additionally, we observed above- random accuracy (0.5) with several model check- points. This indicates that the evaluated models exclusively rely on the presence of superficial sta- tistical artefacts without learning the underlying rules of the tasks. Lexical and syntactic patterns such as token distributions, statement lengths, and discriminative conditions that are disproportion- ately associated with particular class of dataset can superficially inflate model performance (Her- lihy and Rudinger, 2021). These statistical artefacts are typically introduced during the annotation (Gu- rurangan et al., 2018) and allow statement-only
9 Experiment Results
9.1 Task 1
Table summarises the performance of the baseline models on Task 1. We report the results from the model checkpoints with the highest Fl score on the test set. BM25, T5, BERT-base, and GPT2 fail to achieve over 0.5 accuracy the test set. The best-performing models the test set were BioBERT, BioMegatron, and RoBERTa-base, each achieving over 0.6 accuracy and Fl score. BERT base reported a significantly lower Fl than the other baselines.

---

0.8
0.8
Single Comparison
0.7
Numerical
0.3
0.5
0 0.4
0.4
0.3
0.2
0.0
ULRI DOULRI
UM25 RoULKla
GPT2
Bio- ChstGPT
BERT BIOBERT
T5
BM25 RoBERD
OPT2
Bio- ChorGPT
Megatron
Megatron
NLI Numerical
(b) Single vs Comparison
Figure 4: Fl score achieved by the baseline models on different test splits.
classifiers achieve significantly above-random results (Poliak et al., 2018; Tsuchiya, 2018). We support the statement-only baseline with qualitative analysis of the dataset. We found that there was no significant difference between the sentence length distributions of the two classes. Additionally, we find 100% overlap in the 15 most common tokens between the two classes in the training and 93% overlap in the test However, several individual tokens were not evenly distributed across classes in the test set. particular the distributions of not, more, group, and cohort. However, these uneven distributions were not present, if not completely reversed in the training set. Therefore these do not entirely explain the above random accuracy.
of the test set. We evaluate the best-performing checkpoints on these categories and report the re- sults in Figure 4. BioBERT reported the highest Fl score for both Numerical and NLI categories, at 0.588 and 0.686 respectively. The performance on the NLI instances is significantly higher than the performance on the numerical instances, for all models other than BM25 and GPT2 which only marginally report the same trend. Biomedical pre- training does not appear provide a significant advantage within either category. Overall these results follow the trends observed prior works, numerical reasoning remains a challenging task for NLI models (Ravichander et al., 2019; Galashov et al., 2019; Peng et al., 2021; Patel et al., 2021).
We conducted in-depth investigation into po- tential artefacts and biases throughout our experi- ments, yielding no definitive evidence of substan- tial statistical biases. Specifically, the statement- only baseline attains an Fl score of 0.626, notably below the majority class baseline, which is 0.66 F1. Given that the entailment task involves binary clas- sification and acknowledging that existing models have propensity for identifying spurious correla- tions in data, can be concluded that the dataset contains minimal such artefacts.
Single vs Comparison. We repeat this experi- ment, this time categorizing by instance type. Com- parison type instances have 2 CTR sections as a premise, effectively doubling the amount of irrele- vant and potentially adversarial information. There- fore, would expect comparison instances to be more challenging than the single type. We report the results of this experiment in Figure 4. Con- trary expectations the majority of the models there was no significant difference in performance across the categories. BioMegatron reports +0.1 F1 on single type instances, and BM25, and T5 report +0.09 and +0.07 Fl score respectively on the comparison type. As T5 failed to learn Task I, the training set, and BM25 is not able capture clues that go beyond lexical overlaps, we hypothesize that this is in part due the model relying on superficial clues to predict the output label. Therefore, in many cases, the evidence is just not being leveraged properly.
9.4 Categorical Analysis
Numerical Reasoning We categorize the in- stances in our dataset into those that require numer- ical or quantitative reasoning and those that require pure NLI. We separate out all instances with state- ments that contain numbers or any of the follow- ing tokens; higher, lower, fewer, more, less, num- ber, same into the Numerical reasoning category. This results in a 62% Numerical 38% NLI split

---

Model
mAP
Model
AEs
Eligibility
Intervention
Results
BM25
0.786
BERT-bac base
0.649
0,554
0.547
0.537
BioBERT
0.619
0.667
0,588
0.628
all Ml--MiniLM-L6-v2
0.777
BioMegatron
0.661
0.576
0.626
0.528
all-distilooherta-v
0.762
BM25
0.158
0,490
0.529
0.574
all-mpnet-base-v2
0.760
GPT2
0.615
0.531
0.523
0.598
sentence-BioBert
0.752
15-base
0.504
0.515
0.500
0.455
RoBERTa
0.667
0.609
0.523
0.460
sentence-tS-base
0.749
GPT turbo
0.700
0.602
0.511
0.612
Table 5: Results from the NLI4CT Task baselines on the test set.
Average Rank
1.86
2.29
2.86
3.00
Table 4: Fl score of the haseline models on the test set, categorized by CTR section.
these tasks, it would progress the field toward the ef- ficient retrieval and synthesis of published literature to support evidence baadence-bbsed medicine. Additionally, would address several important NLU challenges related to numerical reasoning, biomedical NLI, and inference over long texts.
CTR Sections. Finally, we categorize the in- stances in our dataset by CTR section and report the results in Table 4. All models show signifi- cant fluctuations in performance across the differ- ent sections. Interestingly, Results and AEs on opposite ends of the rankings, despite arguably containing the most similar types of information. Additionally, Eligibility ranked 2nd despite hav- ing the highest average number of tokens.
Our baselines outline the weaknesses of existing NLI models, particularly with regard to numeri- cal inference. We tested 6 SOTA NLI models Task and reported a maximum Fl score of 0.644. Additionally, we show that the limiting factor for current models is not evidence selection, but rather leveraging that evidence, and using numerical and biomedical inference for predictions.
9.5 Task 2
For Task 2. the evidence selection task, we gen- erate a baseline using SOTA general purpose transformer-based ranking models (Reimers, 2020), namely, DistilRoBERTa (Sanh et al., 2019), Mp- net (Song et al., 2020), and MiniLM (Wang al., 2020). We additionally test sentence-t5-bast (Ni et al., 2021), sentence-Biooe BioBERT (Deka and Jurek- Loughrey, 2021), and BM25. We embed the facts in the CTR sections, and the statements using these models and compute the cosine similarity. We eval- uate using the MAP @ K (Liu, 2011), with K total number of sentences, and report the results in Table 5. There is significant difference between the mAP scores of the models, with BM25 produc- ing the highest mAP score, indicating that task is still challenging for dense retrieval models, even with biomedical pre-training.
In future works, we plan generate subset of adversarial instances via controlled set of pertur- bations of words/numbers in statements, Allowing us to perform interventional and causal analysis of models' behavior for numerical and multi-hop inference (Stolfo et al., 2022), in the context biomedical applications, as well as to extend the size of the dataset. Our corpus, competition leader- board, code, and website are available CodaLab and GitHub.
11 Ethical Statement
Annotation. The domain expert annotators are all authors of this paper and were therefore not compensated. The three volunteers for the reclassi- fication were recruited through personal referrals and contributed their efforts on pro bono basis.
10 Conclusions & Future work
We present 2 tasks, textual entailment of claims over clinical trial reports (CTRs), and extracting evidence from CTRs support contradict these claims. We provide corpus of 2400 expert- annotated instances to test and train models on these tasks. models could be developed to solve
Data Usage. To the best of our knowledge, we entirely in line with the ClinicalTrials..en terms of service. We acknowledge ClinicalTrials, gov the data source and explicitly specify the date of data retrieval. Additionally, we provide a detailed account of the modifications made to the dataset.

---

12 Limitations
Pritam Deka and Anna Lurek-Loughrey 2021. Unsuper- vised keyword combination query generation from online health related content for evidence-baser fact checking. In The 23rd International Conference on Information Integration and Web Intelligence, pages 267-277.
The most significant limitation of this work is that despite the fact that we are dialoguing with mo- tivational scenario for medicine, these models are not fit for medical application, as this would require a technology clinical trial, generalization studies and regulatory assessment. Despite this, we believe that developing benchmarks such as NLI4CT rep- resents necessary step toward the development of models capable of clinical decision-making Additionally, our evaluation does not include an interventional study, meaning we do perform penurbatioos//neeeventioss on the test instances to verify that the models learn the underlying causal structure of the tasks. We plan to extend NLI4CT with interventional studies in future work.
Jacob Devlin, Ming-Wor Wei Chang. Kenton Lee, and Kristina Toutanova, 2018. Bert: Pre-training of deep bidirectional transformers for language understand- ing. arXiv preprint 1xXiv:810004880)
Jay De Younng, Eric P. Lehman, Benjamin Nye, lain James Marshall, and Byron C. Wallace. 2020. Evidence inference 2.0: More data, better models. ArXiv, abs/2005.04177.
Ali Emami, Paul Trichelair, Adam Trischler, Ka- heer Suleman, Hannes Schulz, and Jackie Chi Kit Cheung. 2018. The knowref coreference corpus: Removing gender and number cues for difficult pronominal anaphora resolution. arXiv preprint urXiv:1811.01747.
Finally, our dataset contains fewer instances than other published biomedical NLI datasets. This is a consequence of the time and resource-intensive nature of expert-level annotation. The size of the training set may be problematic for large models.
Hui Fang, Tao Tao, and ChengXiang Zhai 2004. A formal study of information retrieval heuristics. In Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval, pages 19-56.
References
Alexandre Galashov, Jonathan Schwarz. Hyunjik Kim, Marta Garnelo, David Saxton, Pushmeet Kohli, S. M. Ali Eslami, and Yee Whye Teh. 2019. Meta- learning surrogate models for sequential decision making. CoRR, abs/1903.1 1907.
Nancy E Avis, Kevin W Smith, Carol L. Link, Gabriel N Hortobagyi, and Edgardo Rivera. 2006. Factors as- sociated with participation in breast cancer treatment clinical trials. / Clin Oncol, 24(12): 1800-1877.
Lisa Grossman Liu, Raymond H Grossman, Elliot G Mitchell, Chunhua Weng. Karthik Natarajan, George Hripesak, and David K Vawdrey. 2021. A deep datahase of medical abbreviations and acronyms for natural language processing. Scientific Data, 8(1):1- 9.
Hilda Bastian, Paul Glasziou, and lain Chalmers. 2010. Seventy-five trials and eleven systematic reviews a day: how will we ever keep up? PLoS medicine, 7())xe000032.
Iz Beltagy, Kyle Lo, and Arman Cohan. 2019, Scibert: A pretrained language model for scientific text. arXiv preprint arXiv: 1903.10677 10676.
Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama. Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. 2021. Domain-specific lan- guage model pretraining for biomedical natural lan- guage processing. ACM Transactiony Computing for Healthcare (HEALTH), 3(1):1-23.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901.
Suchin Gururangan, Ana Marasović. Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A Smith. 2020. Don't 't stop pretraining: adapt language models to domains and tasks. arXiv preprint arXiv:2004 10964.
Leonardo Campillos-Laame Ana Valverde- Adrián Capllonch-Carrie and Antonio Moreno- Sandoval. 2021. A clinical trials corpus annotated with umls entities to enhance the access to evidence- based medicine. BMC medical informatics and deci- sion making, 11(1)
Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel R Bowman, and Noah A Smith. 2018. Annotation artifacts in natural language inference data. arXiv preprint 1803
Wenhu Chen, Hongmin Wang, Jianshu Chen, Yunkai Zhang, Hong Wang, Shiyang Li, Xiyou Zhou, and William Yang Wang. 2019. Tabfact: A large-scale dataset for table-based fact verification. CoRR, abs/1909.02164.
Tianyong Hao, Alexander Rusanov, Mary Regina Boland, and Chunhua Weng. 2014. Clustering clin- ical trials with similar eligibility criteria features. Journal of biomedical informatics, 52:1 12-120.

---

Christine Herlihy and Rachel Rudinger. 2021. Mednli is not immune: Natural language inference ar- tifacts in the clinical domain. arXiv:2106.0149)
Jiao Li, Yueping Sun, Robin Johnson, Daniela Sci- aky, Chih-Hsuan Wei, Robert Leaman, Allan Peter Davis, Carolyn J Mattingly, Thomas C Wiegers, and Zhiyong Lu. 2016. Biocrcative cdr task corpus: a resource for chemical disease relation extraction. Database, 2016.
arXiv preprint
Danqing Huang, Shuming Shi, Chin- Yew Lin, Jian Yin, and Ving Ma. 2016. How well do computers solve math word problems? large-scale dataset con- struction and evaluation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 887-896.
Hao Liu, Yuan Chi, Alex Butler, Yingeheng Sun, and Chunbua Weng. 2021. A knowledge base of clin- ical trial eligibility criteria. Journal of biomedical informatics, 10:101377.
Peter Jansen, Mokanarangan Thayaparan, Marco Valentino, and Dmitry Ustalov. 2021. Texigraphs 2021 shared task on multi-hop inference for expla- nation regeneration. In Proceedings of the Fifteenth Workshop on Graph-Bace Based Methods for Natural Lan- guage Processing (Tex:Graphs-15)
Tie- Yan Liu, 2011. Applications learning to rank. Learning to Rank for Information Retrieval, page 181-191.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint 197.
Peter Jansen and Dmitry Ustalov. 2019. Textgraphs 2019 shared task on multi-hop inference for explana- tion regeneration. In Proceedings of the Thirteenth Workshop on Graph- Baast- Methods for Natural Lan- guage Processing (TexxGraphs-I 13), pages 63-77.
Riccardo Miotto and Chunhua Weng. 2015. Case-based reasoning using electronic health records efficiently identifies eligible patients for clinical trials. Journal of the American Medical Informatics Association, 22
Min Jiang, Yukun Chen, Mei Liu, S Trent Rosenbloom, Subramani Mani, Joshua C Denny, and Hua Xu. 2011 I. A study of machine-learing based approaches to ex- tract clinical entities and their assertions from dis- charge summaries. Journal of the American Medical Informatics Association, (85))601-606
Sungrim Moon, Bridget McInnes, and Genevieve B Melton. 2015. Challenges and practical approaches with word sense disambiguation of acronyms and abbreviations in the clinical domain. Healthcare informatics research, 21(1):35-22.
Antonio J Jimeno- Yepes, Bridget T McInnes, and Alan R Aronson. 2011. Exploiting mesh indexing in medline to generate a data set for word sense disam- biguation. BMC bioinformatics, 12(1):1-14.
Jianmo Ni, Gustavo Hernández Ábrego, Noah Con- stant, Ji Ma, Keith B. Hall, Daniel Cer, and Yin- fei Yang. 2021. Sentence-t5: Scalable sentence en- coders from pre-trained text-to-text models. CoRR. 0002200.08877..
Qiao Jin, Jinling Liu, and Xinghua Lu. 2019. Deep contextualized biomedical abbreviation expansion. arXiv preprint arXiv: 1 1906.03688 03360.
Arkil Patel, Satwik Bhattamishra, and Navin Goyal. 2021. Are nlp models really able to solve simple math word problems? arXiv preprint 11X1v:210..07191.
Svetlana Kiritchenko, Berry De Bruijn, Simona Carini, Joel Martin, and Ida Sim. 2010. Exact: automatic extraction of clinical trial characteristics from journal publications. BMC medical informatics and decision making, 10(1):1-17.
Shuai Peng, Ke Yuan, Liangcai Gao, and Zhi Tang. 2021. Mathbert: A pre-trained model for math- ematical formula understanding. arXiv preprint arXiv:2105.00377
Jan A Kors, Simon Clematide, Saber A Akhondi, Erik M Van Mulligen, and Dietrich Rebholz-Schuhman.. 2015. A multilingual gold-standard corpus for biomedical concept recognition: the mantra gse. Journal of the American Medical Informatics Associ- ation, 22(5):948-956
Ahmad Pesaranghader. Stan Matwin, Marina Sokolova. and Ali Pesaranghader. 2019. deepbiowsd: effective deep neural word sense disambiguation of biomedical text data. Journal of the American Medical Informat- ics Association, 26(5):438 446.
Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So. and Jaewoo Kang- 2019. Biobert: a pre-trained biomedical language representation model for biomedical text mining. CoRR, abs/1901.08746.
Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. 2018 Hypothesis only baselines in natural language infer- ence. arXiv preprint wwXiv:8805.0144
Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2020. Biobert: pre-trained biomedical language representation model for biomedical text mining. Bioinformatiic, (6(4)):23441222
Alec Radford, Jeffrey WU, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAl blog, 1(8)9

---

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter Liu, et al. 2020. Exploring the limits of transfer learning with unified text-to-text trans- former. J. Mach. Learn. Res., 21(140):1-61
support systems: benefits, risks, and strategies for success. NPJ digital medicine, 3(1):1-10.
Mokanarangan Thayaparan, Marco Valentino, and André Freitas. 2021. Explainable inference grounding abstract chains for science questions. In Findings of the Association for Computational Lin- guisties: ACL-IJCNLP 2021, pages 1-12, Online. Association for Computational Linguistics.
Abhilasha Ravichander, Aakanksha Naik, Carolyn Pen- stein Rosé, and Eduard H. Hovy. 2019, EQUATE: benchmark evaluation framework for quantita- tive reasoning in natural language inference. CoRR,
abs/1901.0377 03735..
Paul Trichelair, Ali Emami, Adam Trischler, Kaheer Suleman, and Jackie Chi Kit Cheung. 2018. How reasonable are common-sense reasoning tasks: A case-study on the winograd schema challenge and swag. arXiv preprint arXiv:1811.01777.
Nils Reimers. 2020.
Pretrained models. https: / //www. , sbert. net//dcs///eteeeeeee http//oos//reeeeaiiia mcdels html. Accessed: 2010-09-30.
Stephen E Robertson, Steve Walker, Susan Jones, Micheline M Hancock-Beaulient Mike Gatford, et al. 1995. Okapi at tree-3. Nist Special Publication Sp, 109:109.
Masatoshi Tsuchiya. 2018. Performance impact caused by hidden bias of training data for recognizing textual entailment, arXiv preprint arXiv::804 08110.
Alexey Romanov and Chaitanya Shivade. 2018. Lessons from natural language inference in the clini- cal domain. arXiv preprint 18X.:v880800672.
Marco Valentino, Mokanarangan Thayaparan, Deborah Ferreira, and André Freitas. 2022a. Hybrid autore- gressive inference for scalable multi-hop explanation regeneration. Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 14033 11411.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, a distilled version of BERT: smaller, faster, cheaper and lighter. CoRR, abs/1910.01108.
Marco Valentino, Mokanarangan Thayaparan, and An- dré Freitas. 2022b. Case-based abductive natural language inference. In Proceedings of the 29th Inter- national Conference on Computational Linguistics, pages 1556-1566 Gyeongju, Republic of Korea. In- ternational Committee on Computational Linguistics.
Benjamin Shickel, Patrick James Tighe, Azra Bihorac, and Parisa Rashidi. 2017. Deep ehr: survey of recent advances in deep learning techniques for elec- tronic health record (ehr) analysis. IEEE journal of biomedical and health informatics, (3(5):589--66)
Hoo-Chang Shin, Yang Zhang. Evelina Bakhturina, Raul Puri, Mostofa Patwary, Mohammad Shoeybi, and Raghav Mani. 2020. Biomegatron: Larger biomedical domain language model. arXiv preprint arXiv:2010.0600..
Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang. and Ming Zhou. 2020. Minilm: Deep self- attention distillation for task-agnostic compression of pre-trained transformers. CoRR, abs/2002. 10957.
Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catan- zaro. 2019. Megatron-Im: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv: 1909.08053.
Yonghui Wu, Jun Xu, Yaoyun Zhang, and Hua Xu. 2015. Clinical abbreviation disambiguation using neural word embeddings. In Proceedings of BioNLP 15, pages 171-176.
12.1 Appendices
lan Soboroff. 2021. Overview of trec 2021. In 30th Text REtrieval Conference. Gaithershurg, Maryland.
12.1.1 Pre-processing details
Kaitao Song. Xu Tan, Tao Qin, Jianfeng Lu, and Tic- Yan Liu. 2020. Mpnet: Masked and permuted pre- training for language understanding. Advances in Neural Information Processing Systems, 33: 16857- 16867.
We extracted all CTR data from ClinicalTriaas.er on the 22nd d December 2022. The Eligibility cri- teria and intervention information retrieved from the Eligibility criteria and Intervention sections. The results information is taken from the Primary Outcome section. The Adverse events section is taken from the Serious Adverse Events section, not to be confused with the adverse events section. We remove or replace symbols or tags produced in the extraction, and segment the CTR data into individual lines. These are the only changes made to the CTR data.
Alessandro Stolfo, Zhijing Jin, Kumar Shridhar, Bern- hard Schölkopf, and Mrinmaya Sachan. 2022. A causal framework quantify the robustness of math- ematical reasoning with language models. arXiy preprint arXiv:2210.122
Reed T Sutton, David Pincock, Daniel C Baumgart, Daniel C Sadowski, Richard N Fedorak, and Karen I Kroeker. 2020. An overview clinical decision

---

12.1.2 Zero-shot GPT 3.5 prompt
prompt "Given section of clinical trial de- scriptions and statement, determine whether the statement logically follows from the sections. If the statement logically follows the sections, you need to return 'Entailment'. If the statement does not logically follow from the sections, you need to return 'Contradiction'. The output should be single word <Entailment> or <Contradiction>. "Statement: + Statement
"Primary Trial: Primary CTR text
"Secondary Trial: " Secondary CTR text
12.2 Tables and Graphs

---

Fl score
Model
Single
Comparison
Model
Numerical
NLI
BERT-base
0.364
0.582
BERT-base
0.493
0.664
BioBERT
0.635
0.619
BioBERT
0.588
0,686
RigMarrt
0.648
0.550
BioMegatron
0.552
0.660
BM25
0.517
BM25
0.468
0.495
GPT-2
0.585
0,550
GPT-2
0.553
0.577
RoBERTa-base
0.569
RoBERTa-base
0.463
0.676
T5-base
0.455
0.527
T5-buse
0.429
0.596
GPT 3.5 turbo
0.627
0.584
GPT 3.5 turbo
0.562
0.635
Table 6: Fl score of the baseline models on the test set, categorized by instance type.
Table 7: Fl score of the baseline models on the test set, categorized by numerical and NLI inference.
Model
F1
Precision
Recall
Accuracy
Epoch
Fl score
Accuracy
BM25
0.622
0.489
0.856
0.480
Pre-truined
0.008
0.500
1
0,666
0.506
BERT-base
0.597
0.569
0.628
0.576
0.527
0.590
RoBERTa-base
0.659
0.506
0.944
0.512
3
0.623
0.572
BioBERT
0.667
0.589
0.768
0.616
4
0.613
0.604
BioMegatron
0.596
0.543
0,660
0.552
0.522
0.578
GPT2
0.581
0.519
0.660
0.524
6
0.485
0.584
T5-base
0.521
0.522
0.520
0.522
0.572
0.590
8
0.575
0.598
Table Results from the NLI4CT Task evidence- only baselinc on the test
0.494
0.590
0.626
0.606
Table BERT-base results on Statement-only test set.
learning
nam layers num attention
BERTT-base uncased
L.00 05
32
768
12
12
tmps:/hggitttttss co ber hase uncased
BioBERT-11.1
1.00-05
32
768
12
12
Trtps hiisslhhoiiggssss en mndimiss bithiessiiiitsss1111
BioMegatron
LU09-05
1024
age http://www........aa.eeeeeeeeeee
GPT2
1.00--05
708
hops hmpps//gggggeeee gpt?
TS-hoa
1.00-005
768
12
hapsithugiiggiesscmmm aut S-bane
Table 6: Additional model implementation details for Task 1.

---

Train -Dev -Test Random
0.8
0.7
F1 score
0.6
0.5
0.4
0.2
0.1
Base model
Epoch
Figure 5: Graph of the BioBERT Fl score by epoch, the test, training, and development
BERT-base
BioBERT
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
3
4
5
6
7
8
9
10
Epochs
Figure 6: Graph of BERT-base F1 score by epoch, the test compared to Bio-BERT.

---

0.08
0.04
Distribution
Entaa Iment
0.01
Number of tokens per statement
Figure 7: Distribution curve of the number of tokens per statement in the test and training set.

---

Annotation ID: 0
Variable prompt:
Intervention
Primary Clinical trial ID: NCT02953860
INTERVENTION 1:
Fulvestrant With Enzalutamide
500mg of Fulvestrant will be given IM on days 1, 15, 28, then every 4 weeks as per standard care (SOC) and 160mg of Enzalutamide will be given, conjunction with Fulvestrant, PO daily.
Fulvestrant with Enzalutamide: 500mg of Fulvestrant will be given IM on days 1, 15, 28, then every 4 weeks as per standard
(SOC) and 160mg of Enzalutamide will be given PO daily. Patients will receive a tumor blopsy at the start of treatment and weeks after the start treatment, with an optional 3rd biopsy at the end treatment.
Secondary Clinical trial ID: NCTO1630933
INTERVENTION 1:
Usual
Usual Care is the comparison Clinic Patients, where there is no change in their standard usual care.
INTERVENTION 2:
BreastCARE Intervention
Intervention Clinic Patients: The participants will answer questions the tablet-PC to calculate their breast cancer risk.
Intervention Patient Report. Once the patient completes the BreastCare Computer survey, the program will immediately generate a personal feedback report containing information about her risk factors and recommendations reduce her risk. This report will be printed and given to the patients before she meets with her doctor.
BreastCARE: The physician will receive a physician report that contains information similar to the patient report.
Using the information above, please write entailment statement below, please include NCT numbers in the statement. You can use the Annotation guidelines if necessary:
Figure 8: Page example annotation form.

---

Entailment statement:
Contradiction statement:
Link to Annotation guidelines
Figure 9: Page 2 of example annotation form.

---

1 Annotation Guidelines
2
Motivation
This aunotation process part project build datasct from clinical data. Clinical trials are type of research that studies new tests and treatments and evaluates their effects on human health outcomes. The clinical trial data in this dutaset is brenst cancer
clinicaltrills gov. process will be used for inference relation task. This will allow further research develop systems that can evaluate clinical trial texts.
generated through
2.1 Resources
separatted CTRs sectioms:
Eligibility criteria: conditions patients
part
the clinical trial.
Intervention: type, dosage, frequency, srudied.
being
Results: Number participants trial, outcome measures, units, and results.
Adverse Events: These and symptoms observed patients during
clinical trial.
Ellighility Criteria
Inctusion:
11ER2 + diagnosis
years age
Excluion:
Presence central
syseem or bruin metastuces.
Interveation
Gemecia bine 1500
surface
ares (HS A) Intravemonsly (IV) over 30 minutes) days 15 of each eyele.
Progression
Number of Participnts: 29 (montss); 10.1
Adverse
Total: 3/99
Neutropnnic
(3.45%)
Peripheral Neuropathy 1/ (3.45%)
Seizure/Syccuge (3.45%)
You will receive CTR section prompt, which pertains to one of the four sections, along with correspond:ng information from both primary secondary trial.
Figure 10: Page of Annotation guidelines.

---

For every pair of trials, you must write statements, an entailment statement, and contradictory statement. When writing statements, you must always refer to the primary and secondary trials "Primary Clinical Trial" and "Secondary Clinical Trial". If the data is illeziblekerroncous please contact with the clinicel trial ID of the data. We provide you with 100 sample annotations help demonstrate this process in sumple data docx.
2.2 Entailment statement annotation
Using the information provided write entailment statement. This statement should make an objuzivvlyy true claim about the content of the prompted section of the trial(s), you have option to write a statemeal about the primary trial alonc or to draw compurison between both primary and secondary trials. Aim produce trivial statements. This meuns that to infer the statement's truth, one must read and comprehend multiple rows of the CTR section. Simple techniques for generating Non trivial statements summarization, comparisons, negatim, relations, inclusion. superlatives, oggregations, and sephrasing, However, this list is not exhaustive. Any statement that demands urderstanding and reasening is acceptable. Annotators are encrrraged
non-repetitive statements, require different types masoning, over different types information. Avoid vague words such 'maybe' 'perhups' ctc, that create uncertainty ambiguity, non-definitive statements will be removed.
2.3 Contradictory Statements
Using the entailment statement you's 'v generated, write contradictoon statement. This statement should make an objectively false claim about the section of the trial(s) if the entuilment statement only about the primary trial, the contradiction statement should also only be ahout the primary trial, conversely if the entailment statement includes statement should be the same. Where possible aim to mointain the original staterenut's style and longth.
of the prompted
primary secondary
contradiction
2.4 Evidence retrieval
For each statement, annctators are asked collect evidence proof true/false of their statement. Please collect evidence using each line individual fact, copy and paste the full line into the evidence section. Multiple lines/facts may be used for evidence. Every statement should have leest one piece evidence supporting its instances
(e.g.. "clinical trial does have roquired to preseut the cutire CTR section evidence.
negation
placebo amm"),
Figure 11: Page 2 of Annotation guidelines.


Reviews:
[{"review_id": "ikmKEwr0Q2", "reviewer": "Reviewer_iVg5", "summary": "", "questions": "Some SOTA LLMs are not tested, for example, GPT-4 or LLaMA. ", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. Questions For The Authors: Some SOTA LLMs are not tested, for example, GPT-4 or LLaMA. \n\nThe main focus of the paper is the construction of a new resource for clinical NLI. Therefore, the experiments aim to establish a set of baselines covering different categories of models (including models with different sizes and architectures). Regarding the category of LLMs, we opted for GPT3.5 Turbo since it has a lower cost than GPT4 and it generally exhibits higher overall performance than LLaMa. Moreover, since the focus of the paper is not to exclusively evaluate the performance of LLMs, and these models are very resource intensive, we decided to leave the evaluation of different and possibly larger models for future work. Moreover, our experiments have already shown that specialized pre-training may be more significant than continually increasing model size (Table 3 \\& 4). \n\n2. A significant number of literature is missing in the paper. Particularly those from Chunhua Weng (Columbia University)'s group. They have been doing clinical trial information extraction and normalization for decades. This reviewer would like to see how different contributions this work make compared to Weng's group.\n\nWe agree that Weng's group leading contributions on NLP for CTs need to be better compared and contrasted against this contribution. Some more significant differences are around the fact that this work is more centered around a clinical NLI task. Another difference is the focus on this task on inferences beyond eligibility criteria (also including inference over results, adverse events, or intervention sections). We will extend the related work section in the camera ready eliciting the contributions and differences regarding Weng's long standing contributions in NLP for CTs.\n\n3. Reproducibility: 3: Could reproduce the results with some difficulty. The settings of parameters are underspecified or subjectively determined; the training/evaluation data are not widely available.\n\nTraining/evaluation data were submitted in supplementary materials, and links to the data were anonymized, as instructed. Parameter settings are displayed in a table in the appendix (Table 6), and are implemented in a standardized manner.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Extract the information from clinical trials report and inference on the patient eligibility is an important task to facilitate patient recruitment for clinical trials. The problem is very significant for clinical and translational research. The dataset released by this paper will be very valuable to the medical informatics community.", "reasons_to_reject": "A significant number of literature is missing in the paper. Particularly those from Chunhua Weng (Columbia University)'s group. They have been doing clinical trial information extraction and normalization for decades. This reviewer would like to see how different contributions this work make compared to Weng's group.", "questions_to_authors": "Some SOTA LLMs are not tested, for example, GPT-4 or LLaMA. ", "paper_topic_and_main_contributions": "This paper defined a novel benchmark, called NLI4CT, for clinical trial reasoning and inference tasks. This benchmark includes two tasks. \nThis paper also released a new corpus with 2400 expert annotated entailment relations.\nThis paper tested and compared 7 SOTA NLI models .", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ZKWBj2KCHf", "reviewer": "Reviewer_6bps", "summary": "", "questions": "A. On the annotation process, how many expert annotators were involved, did they have a balanced load on the annotation, how was the annotation guide developed, and what did the annotators encounter the main problems and disagreements?\n\nB. Given that the resource is for NLI model testing only... How much human effort and time would it take to convert this resource into a complete training dataset to generate a real application?  \n\n(the answers given in the rebuttal are appropriate).", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. The corpus compilation and its annotation is not fully described. Please see question A below. This point is relevant because the paper should address potential biases and limitations in the annotation process.  A - On the annotation process, how many expert annotators were involved, did they have a balanced load on the annotation, how was the annotation guide developed, and what did the annotators encounter the main problems and disagreements?\n\nIn total there were 4 expert annotators, with one expert performing the majority of the annotations. The annotation guide was carefully refined after the annotation of 100 Pilot examples from the main expert annotator to make the whole annotation process as efficient as possible, we will publish the annotation guide in the appendix of the camera ready. There were no significant problems or disagreements throughout the annotation process, as shown in lines 247-251, we achieved a very high rate of agreement (Cohen’s kappa of 0.83) and an 85.66\\% average accuracy with respect to the gold labels.\n\n\n2. Potential Biases: The article should address any potential biases introduced during the data collection and annotation. Biases could impact the generalizability and reliability of the findings.\n\nAs explained in the paper (Lines 476-488) we investigated several different possible artefacts/biases in our experiments, finding no conclusive evidence of strong statistical biases. Specifically, the statement-only baseline achieves an F1 of 0.626 which is significantly below the majority class baseline (0.66 F1). Since the entailment task is a binary classification task and considering that existing models are well-known for finding spurious correlations in the data, we can conclude that the presence of such artifacts in the dataset is very minimal.\n\n\n3. The list of limitations acknowledged by the authors: the models developed are not fit for direct medical application; the evaluation also lacks an interventional study to verify if the models learn the underlying causal structure of the tasks; and the dataset contains fewer instances than other published biomedical NLI datasets.\n\nWe believe that developing benchmarks such as NLI4CT represents a necessary step toward the development of models capable of clinical decision-making. However, at the same time, we acknowledge that NLI4CT alone is not enough for real-world applications and it has to be complemented with further analyses, the development of models that are fit for medical applications, generalization studies, and regulatory compliance among other things. Additionally, with respect to the size of our dataset, we address the future steps we will be taking to increase the size of our dataset in the conclusion of our paper.\n\n4. In summary, the main reason for rejecting the paper would be that the scope of the resource is limited to testing NLI models.\n\nWe believe that developing benchmarks such as NLI4CT represents a necessary step toward the development of models capable of clinical decision-making, and this is a rich and complex domain, with enough depth to warrant a paper.  Additionally, we believe that we achieved all of our aims, within this scope.\n\n5. They also cover in a table the comparison with other similar resources. References to similar resources on CTR in other languages need to be included. For example: Kors JA, Clematide S, Akhondi SA, van Mulligen EM, Rebholz-Schuhmann D. A multilingual gold-standard corpus for biomedical concept recognition: the Mantra GSC. J Am Med Inform Assoc. 2015;22(5):948–56. Campillos‑Llanos et al. (2021) A clinical trials corpus annotated with UMLS entities to enhance the access to evidence‑based medicine. BMC Med Inform Decis Mak (2021) 21:69 https://doi.org/10.1186/s12911-021-01395-z\n\nWe will add these references in the related work section of the camera-ready as different tasks in a similar domain (thank you for pointing them out).  \n\n6. Given that the resource is for NLI model testing only... How much human effort and time would it take to convert this resource into a complete training dataset to generate a real application?\n\nWe believe that developing benchmarks such as NLI4CT represents a necessary step toward the development of models capable of clinical decision-making. To convert this resource into a complete training dataset to generate a real application NLI4CT has to be complemented with analyses which are focused on their clinical safety, going through a validation process which typically transcends NLP methodologies (i.e. a technology clinical trial). While the size of the dataset that would be required to deliver a model for clinical use is a factor, the most important factors may be not related to the dataset (e.g. quantifying uncertainty). This dataset is a cornerstone enabler towards the construction of these models. \n\n7. Describe how intellectual property (copyright, etc) was respected in the data collection process. Describe how crowd workers or other annotators were fairly compensated and how the compensation was determined to be fair.\n\nThe group of annotators that took part in the annotation process are all authors of the paper and were therefore not compensated. The three annotators for the reclassification were volunteers recruited within the expert network of the authors. These contributed their efforts on a pro bono basis. We will elicit these points in the camera ready.\n\n8. Describe how intellectual property (copyright, etc) was respected in the data collection process.\n\nTo the best of our knowledge, we are entirely in line with the clinicaltrials.gov terms of service, as ClinicalTrials.gov data is available to all requesters, both within and outside the United States In the dataset documentation and in the paper we attribute the source of the data as ClinicalTrials.gov and clearly state the modifications made to the content of the data, along with a complete description of the modifications (Lines 145-172, 199-261). We can elicit these points further in the camera ready.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The most important contributions of the paper:\n\n1. NLI4CT Resource itself. NLP needs good resources annotated by experts.\n\n2. Addressing Biomedical NLI Challenges: CTRs are technical and complex documents challenging natural language inference. The paper highlights the importance of addressing challenges related to numerical reasoning, biomedical NLI, and processing long texts.\n\n3. Baseline Performance Comparison: The paper presents test results with 6 state-of-the-art NLI models on the NLI4CT corpus. These results highlight the current limitations of existing NLI models, especially regarding numerical reasoning. The performance comparison is a benchmark for future improvements and advancements in the field.\n\n4. Availability of Resources: The authors provide public access to their NLI4CT corpus, a competition leaderboard, a website, and the code necessary to replicate the baseline experiments. \n\nOverall, the paper presents contributions to the NLP community in the biomedical field by addressing the challenge of natural language inference in clinical trial reports. The development of the NLI4CT corpus and the findings on the limitations of current models provide a solid basis for future research and improvements in this area. The availability of resources also fosters collaboration and progress in the field.", "reasons_to_reject": "1. The corpus compilation and its annotation is not fully described. Please see question A below. This point is relevant because the paper should address potential biases and limitations in the annotation process.\n\n2. Potential Biases: The article should address any potential biases introduced during the data collection and annotation. Biases could impact the generalizability and reliability of the findings. \n\n3. The list of limitations acknowledged by the authors: the models developed are not fit for direct medical application; the evaluation also lacks an interventional study to verify if the models learn the underlying causal structure of the tasks; and the dataset contains fewer instances than other published biomedical NLI datasets.\n\nIn summary, the main reason for rejecting the paper would be that the scope of the resource is limited to testing NLI models. \n\n(the answers given in the rebuttal are appropriate).", "questions_to_authors": "A. On the annotation process, how many expert annotators were involved, did they have a balanced load on the annotation, how was the annotation guide developed, and what did the annotators encounter the main problems and disagreements?\n\nB. Given that the resource is for NLI model testing only... How much human effort and time would it take to convert this resource into a complete training dataset to generate a real application?  \n\n(the answers given in the rebuttal are appropriate).", "paper_topic_and_main_contributions": "The paper falls under \"new data resources\" and \"NLP engineering experiment\" topics. It discusses the development of a novel resource called NLI4CT for advancing research on Natural Language Inference (NLI) for reasoning on clinical trial reports (CTRs). \n\nThe main hypothesis is that Clinical trial reports (CTR) amassed over the years contain indispensable information for developing personalised medicine. However, it is practically infeasible to manually inspect over 400,000+ clinical trial reports to find the best evidence for experimental treatments. Natural Language Inference (NLI) offers a potential solution to this problem by allowing the scalable computation of textual entailment. However, existing NLI models perform poorly on biomedical corpora, and previously published datasets must capture the full complexity of inference over CTRs.\n\nThe contribution of the work is to present a novel resource to advance research on NLI for reasoning on CTRs. The resource applies to two tasks on breast cancer: determining the inference relation between a natural language statement and a CTR and retrieving supporting facts to justify the predicted relation.  They provide a corpus of 2400 expert-annotated instances to train and test models on these tasks. In addition, the paper provides baselines on this corpus to expose the limitations of existing NLI models, with six state-of-the-art NLI models achieving a maximum F1 score of 0.627. The authors claim this is the first attempt to design a task covering the interpretation of full CTRs. They make the corpus, competition leaderboard, website and code available to replicate the baseline experiments.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "Yes", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Overall the paper is well written. There is too much information of different kinds, but it is well distributed in the article's structure."}, "extra_text": {"Missing_References": "The previous literature review is correct and up to date. The authors highlight the papers that have supported the need for more efficacy of neural models in quantitative reasoning and numerical operations within the NLI. \n\nThey also cover in a table the comparison with other similar resources. \n\nReferences to similar resources on CTR in other languages need to be included. For example: \n\n* Kors JA, Clematide S, Akhondi SA, van Mulligen EM, Rebholz-Schuhmann\nD. A multilingual gold-standard corpus for biomedical concept recognition:\nthe Mantra GSC. J Am Med Inform Assoc. 2015;22(5):948–56.\n\n* Campillos‑Llanos et al. (2021) A clinical trials corpus annotated with UMLS entities to enhance the access to evidence‑based medicine. BMC Med Inform Decis Mak (2021) 21:69 https://doi.org/10.1186/s12911-021-01395-z\n\n\n(authors would include those references in the final version).", "Justification_for_Ethical_Concerns": "Following the Ethics Review questions for papers presenting new datasets, the authors should address these questions in their paper:\n\n1. Describe how intellectual property (copyright, etc) was respected in the data collection process.\n\n2 Describe how crowd workers or other annotators were fairly compensated and how the compensation was determined to be fair."}}, {"review_id": "NO92BMhkpn", "reviewer": "Reviewer_hG7J", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. The paper does not report the performance of several recent best-performing biomedical models including PubMedBERT and BioLinkBERT and therefore lack of more domain-specific models evaluation.\n\nThe aim of the experiments is to establish a set of baselines for NLI4CT covering different categories of models (including models with different sizes and architectures) and not to exclusively focus on domain-specific models. Regarding this category, we decided to evaluate BioBert and BioMegatron as representative approaches in the research community and leave a more in depth evaluation of domain-specific models for future research. Moreover, it is important to notice that our experiments also cover larger models (e.g. GPT 3.5) that have demonstrated to exceed the performance of bert-based models on a range of previous domain-specific tasks.\n\n2. Statements-only baseline revealed the presence of artefacts, which is consistent with existing NLI datasets where hypothesis-only performs well, including MedNLI. It would be beneficial to identify and reduce shortcuts exploited by the models to enhance the overall robustness and reliability of the results.\n\nAs explained in the paper (Lines 476-488) we investigated several different possible artefacts in our experiments, finding no conclusive evidence of strong statistical biases. Specifically, the statement-only baseline achieves an F1 of 0.626 which is significantly below the majority class baseline (0.66 F1). Since the entailment task is a binary classification task and considering that existing models are well-known for finding spurious correlations in the data, we can conclude that the presence of such artifacts in the dataset is very minimal.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The paper proposes a novel NLI dataset focused on CTRs with an evidence selection task. The task has the potential to contribute to achieving more transparent and explainable predictions, which is highly needed in this domain.\n\n* The paper is easy to follow, with plenty of examples and detailed explanations.\n\n* The categorizations in NLI4CT allow researchers to do more fine-grained analysis (Results Vs. Intervention, Single Vs. Comparison).\n\n\n", "reasons_to_reject": "\n* The paper does not report the performance of several recent best-performing biomedical models including PubMedBERT and BioLinkBERT and therefore lack of more domain-specific models evaluation. \n* Statements-only baseline revealed the presence of artefacts, which is consistent with existing NLI datasets where hypothesis-only performs well, including MedNLI. It would be beneficial to identify and reduce shortcuts exploited by the models to enhance the overall robustness and reliability of the results.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper introduces a novel binary biomedical NLI benchmark, which utilizes clinical trial reports, specifically focusing on breast cancer. Additionally, it proposes an evidence retrieval task to support the predicted relation (entailment or contradiction). In this paper, each clinical trial report is divided into four sections. These individual sections serve as premises, whereas hypotheses are written by experts. Both tasks provide the means to test the domain knowledge and various capabilities of the models, including numerical and commonsense reasoning. The authors provide several baselines for the proposed benchmark.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "\n* Inconsistent capitalization in Table 1, such as \"neutral\".\n"}, "extra_text": {}}]

Author rebuttal:

Meta-review: As a reviewer indicates this paper defined a novel benchmark, called NLI4CT, for clinical trial reasoning and inference tasks. This benchmark includes two tasks. This paper also released a new corpus with 2400 expert annotated entailment relations. This paper tested and compared 7 SOTA NLI models.

The main reasons to accept the paper are the following ones:


	Extracting the information from clinical trials report and inference on the patient eligibility is an important task to facilitate patient recruitment for clinical trials. The problem is very significant for clinical and translational research. The dataset released by this paper will be very valuable to the medical informatics community.

    NLI4CT Resource itself. NLP needs good resources annotated by experts.

    Addressing Biomedical NLI Challenges: CTRs are technical and complex documents challenging natural language inference. The paper highlights the importance of addressing challenges related to numerical reasoning, biomedical NLI, and processing long texts.

    Baseline Performance Comparison: The paper presents test results with 6 state-of-the-art NLI models on the NLI4CT corpus. These results highlight the current limitations of existing NLI models, especially regarding numerical reasoning. The performance comparison is a benchmark for future improvements and advancements in the field.

    Availability of Resources: The authors provide public access to their NLI4CT corpus, a competition leaderboard, a website, and the code necessary to replicate the baseline experiments.
	
    The paper is easy to follow, with plenty of examples and detailed explanations.

    The categorizations in NLI4CT allow researchers to do more fine-grained analysis (Results Vs. Intervention, Single Vs. Comparison).


The main reasons to reject the paper are the following ones:

 
	Some references are missing in the paper. 

	The corpus compilation and its annotation is not fully described. The paper should address potential biases and limitations in the annotation process.

    	Potential Biases: The article should address any potential biases introduced during the data collection and annotation. Biases could impact the generalizability and reliability of the findings.

    	The list of limitations acknowledged by the authors: the models developed are not fit for direct medical application; the evaluation also lacks an interventional study to verify if the models learn the underlying causal structure of the tasks; and the dataset contains fewer instances than other published biomedical NLI datasets.

	The paper does not report the performance of several recent best-performing biomedical models including PubMedBERT and BioLinkBERT and therefore lack of more domain-specific models evaluation.

Despite the reasons to reject the paper, reviewers agree that the paper is sound and strong. This resource would help make advances in the field.

If the paper is accepted, the final version should be elaborated taking into account the comment by reviewers and by the ethics committee.
Decision: Accept-Main