<figure><img src="image_1.png" /></figure>

HAL
open science

# Assessing the Difficulty of Inference Types in Natural Language Inference for Clinical Trials

Mathilde Aguiar, Pierre Zweigenbaum, Nona Naderi

<figure><img src="image_2.png" /></figure>

## To cite this version:

Mathilde Aguiar, Pierre Zweigenbaum, Nona Naderi. Assessing the Difficulty of Inference Types in Natural Language Inference for Clinical Trials. 2026. hal-05533706

HAL Id: hal-05533706

https://hal.science/hal-05533706v1

Preprint submitted on 2 Mar 2026

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_3.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
# Assessing the Difficulty of Inference Types in Natural Language
Inference for Clinical Trials

Mathilde Aguiar, Pierre Zweigenbaum, Nona Naderi

Université Paris-Saclay, CNRS, Laboratoire Interdisciplinaire des Sciences du Numérique
91400, Orsay, France
{mathilde.aguiar, pierre.zweigenbaum, nona.naderi}@lisn.fr

## Abstract

Large Language Models (LLMs) achieve competitive results on Natural Language Inference when applied to clinical trials; however, it is not yet clear on which type of inference LLMs perform well or poorly. We address this by proposing new supplementary annotations to the existing NLI4CT dataset on the types of inferences observed in clinical trials. Our dataset supplements NLI4CT with a total of 1,851 new annotations using our carefully crafted guidelines for 17 types of inferences. To investigate how the inference types impact the performance of LLMs, we prompt Flan-T5, Llama, Mistral, and Qwen and investigate their performance using our newly annotated dataset. We found that logical inferences have a negative impact on Mixtral, Qwen-7B, and Qwen-14B's overall performance, while numerical inferences have a negative impact on Flan-T5-XL and Mixtral. Further analysis shows that understanding the CTR's structure by itself remains challenging for MMed-Llama-3. Other parameters, such as the number of inference types involved or the type of section used in the premise, also impact models' performance. Our code and dataset are publicly available on GitHub.

**Keywords:** Natural Language Inference, Clinical trials, Large Language Models.

## 1. Introduction

Large Language Models (LLMs) often obtain high accuracy or F1-score when evaluated on Natural Language Understanding tasks (Fourrier et al., 2024). They tend to outperform encoder-only architectures, such as BERT-like (Devlin et al., 2019) models, traditionally used for these discriminative tasks. Despite achieving competitive results, it is often not clear on what elements the model is relying on to make its prediction and what type of knowledge it uses. Natural Language Inference (NLI) consists of determining whether a statement can be inferred from a given premise. The possible outcomes are either *entailment*, *contradiction*, or *neutral*. This task can be quite challenging as the model needs to identify pieces of evidence in the premise and compare them to the statement to determine the inference relation. Often, the entailment relation has to be a multi-hop process, meaning that the model must perform several sub-inferences to deduce the final relation. Models need to understand links between concepts and use previously acquired knowledge, and can not only rely on words' surface forms to deduce the inference relation (Mahowald et al., 2024). We hypothesize that these sub-inferences involve different kinds of knowledge, some of which might be challenging to deal with for LLMs, especially in a domain-specific setting. We test this hypothesis in the clinical trials domain, using the NLI4CT dataset (Jullien et al., 2023a). This dataset applies NLI to clinical trials, covering cases where entailment is tested from their results, adverse events,

interventions or eligibility criteria sections.

We define 17 different inference types observed in NLI4CT, clustered in 6 categories. To do so, we manually annotate NLI4CT's test set and prompt 8 LLMs, pretrained on the medical or the general domain, and using 2 types of architectures: decoder-only models and Sequence-to-Sequence models. Our results show that Flan-T5-xl, Mistral, Qwen-7B and 14B are sensitive to the presence of certain inference types, like numerical and logical inferences, affecting the performance negatively when this type of inference is present.

The contributions of the paper are the following: (i) we propose an annotation scheme for inference types for clinical trials; (ii) we annotate the NLI4CT dataset according to this scheme; (iii) we analyze the performance of 8 LLMs on NLI4CT according to these inference types. (iv) finally, we perform an error analysis. Our code and dataset are publicly available on GitHub.<fnref n="1" />

## 2. Related Work

Dagan et al. (2006) defines the task of NLI to be “*the task of deciding, given two text fragments, whether the meaning of one text is entailed (can be inferred) from another text*”. This task has been studied and applied in various domains. However, there is no universal definition of the existing different inference types, since defining inference types depends on the studied domain and research purpose (Yan et al., 2020). In the following section, we propose

<footnote n="1">¹Github repository</footnote>
a review of existing schemes for inference types
definition.

**2.1. Annotating Inference Types**

Previous studies have defined different inference
schemes specially tailored for their research focus
and application domains.

Yan et al. (2020) creates a schema to focus on
linguistic aspects of inference types in the domain
of opinion mining. The authors focused on provid-
ing annotations on whether an inference is present
and, if so, which type between the following: Log-
ical, Pragmatic, Lexical, Enunciative, and Discur-
sive. Their aim was not to provide an analysis of
models' performance on the NLI task but to ana-
lyze the different types of inference present in their
corpus.

Instead of defining inference type schemes to predict which type of inference is present in a corpus, other studies have defined schemes to analyze the performance of language models on different inference types. These methods allow for evaluating models' reasoning steps instead of solely focusing on global models' accuracy or similar metrics, and to know which kind of knowledge the model is missing. To address this issue, a few annotation schemes (Nie et al., 2020; Joshi et al., 2020; Williams et al., 2022) have been proposed to assess a more fine-grained evaluation of models' performance on inference types (also called reasoning types). Nie et al. (2020) release the ANLI dataset to propose a novel challenging NLI dataset to The authors define the following reasoning types: Numerical & Quant., Reference & Names, Standard, Lexical, Tricky, Reasoning & Facts, Quality. Later on, Williams et al. (2022) reuse ANLI to perform a more in-depth analysis by defining 40 different fine-grained types, clustered in Numeral, Basic, Reference, Tricky, Reasoning, and Imperfections categories. TaxiNLI (Joshi et al., 2020) define a similar taxonomy with the Linguistic, Logical, Knowledge top-level categories. In the clinical trials domain, NLI4CT provides supplementary annotations by separating the Numerical inferences from the rest of the instances.

These schemes were mainly designed for
general-domain applications, only one study
(NLI4CT) addressed the clinical trials domain.
In addition, these studies investigated Masked-
Language Models, whereas our study focuses on
Large Sequence-to-Sequence and decoder-only
models. Sec. 3.2 gives a detailed comparison of
these annotation schemes with the one we pro-
pose.

**2.2. NLI4CT Dataset Description**

The NLI4CT corpus consists of a collection of En-
glish breast cancer Clinical Trial Reports (CTRs)
taken from clinicaltrials.gov. The NLI4CT task ap-
plies NLI to clinical trials with several use cases,
such as checking whether a patient complies with
the trial's eligibility criteria or whether a claim can
be deduced from the trial's results. A premise is
composed of a whole section of a CTR that can
either be the Eligibility Criteria, the Intervention, the
Results, or the Adverse Events. NLI4CT comprises
two kinds of instances: single, where only one CTR
is involved to perform the inference, and compari-
son, where two CTRs are needed to be compared.
A statement is one or two sentences long and has
been created artificially. By analyzing the state-
ment and the premise together, a model should
predict whether the statement entails or contradicts
the premise. The original dataset is balanced be-
tween Entailment and Contradiction labels. The
task depends on several kinds of inference, includ-
ing biomedical reasoning, commonsense reason-
ing, and numerical reasoning.

3. Methodology

In this study, we first systematically investigate the
types of inference and knowledge involved in the
inference process by defining an inference types
schema, annotating the NLI4CT dataset, and then
examining which types of inference remain chal-
lenging for LLMs. We first present our annotation
process (Sec. 3.1), then provide a definition for
each inference type (Sec. 3.2), and evaluate the
models' performance on them (Sec. 3.3).

**3.1. Annotation Process**

We define and identify the different categories of
inference types in the NLI4CT dataset that are
needed to solve the inference relation. The goal is
to define labels that cover all observed inference
types, being both fine-grained enough and with
minimal overlap between them to prevent annotator
disagreements. We follow an incremental process
to define the different inference type labels, drawing
inspiration from existing relevant ones in the litera-
ture to create our own labels and adapt them to our
use case. We sample a first subset of 10% from
NLI4CT's test set, ensuring that these 50 instances
are representative of the full test set in terms of
Entailment/Contradiction and Single/Comparison
ratios. We use this first subset as a base to de-
fine our labels. We start by drawing super-classes
of inference types (see Sec 3.2): Task related ex-
pression, Logical, Generic knowledge, Biomedical,
Numerical, and lastly Typo/Error. Then, we de-
fine more fine-grained categories (see Fig. 3) using
the examples contained in the subset we sampled. These fine-grained labels are the ones used to annotate the test set. Each instance can be labeled with one or more inference types. We asked 3 annotators, all NLP researchers and authors of this paper, to produce the annotations. After the first round of annotations, the annotators discussed the labels and revised the annotation guidelines. We conducted a second round of annotations, during which the annotators worked independently on a subset of 50 additional instances using the final annotation guidelines<fnref n="2" />. After the second round of independent annotations, the annotation conflicts were discussed and resolved. Tab. 1 presents the final inter-annotator agreement using Krippendorff's alpha for this subset.

<table><thead><tr><th>Pair</th><th>α</th></tr></thead><tbody><tr><td>A1 vs A2</td><td>0.67</td></tr><tr><td>A1 vs A3</td><td>0.47</td></tr><tr><td>A2 vs A3</td><td>0.45</td></tr><tr><td>Average</td><td><strong>0.53</strong></td></tr></tbody></table>

(a) Pairwise inter-annotator agreement using Krippendorff's α.

<table><thead><tr><th>Inf. type</th><th>α</th></tr></thead><tbody><tr><td>TR</td><td>1.00</td></tr><tr><td>Num</td><td>0.80</td></tr><tr><td>Bio</td><td>0.59</td></tr><tr><td>GK</td><td>0.37</td></tr><tr><td>Log</td><td>0.75</td></tr><tr><td>T/E</td><td>1.00</td></tr><tr><td>Average</td><td><strong>0.75</strong></td></tr></tbody></table>

(b) Average Krippendorff's' α per inference type between the 3 annotators.

<caption>Table 1: Inter-annotator agreement measures.</caption>

We achieve an average score of $α = 0.53$, indicating that producing annotations for our task remains challenging. However, annotators A1 and A2 obtain a score of 0.67, indicating a fair agreement. Labels, such as *Task-related expression* and *Typo/Error*, were quite straightforward to annotate. On the other hand, *Generic knowledge* was much harder to define, which led to a poorer agreement. This agreement is also due to the lengthy premises and statements (with an average length of 22 words for the statements, 125 words on average, and up to 1388 words for the premises) that make them difficult to analyze and capture all subtleties. The remaining test set was annotated by annotator A1, resulting in a total of 1,851 annotations for 500 statements. Fig. 1 displays the corpus statistics. Fig. 2 presents the correlation matrix between the different inference types. Annotators were also asked to re-annotate the dataset in terms of *Entailment* and *Contradiction*. Annotators identified a few mistakes and achieved an $α$ of 0.97 with the original NLI4CT's annotations. While the original test set was balanced between *Entailment* and *Contradiction*, we obtain slightly more *Contradiction* instances (253) than *Entailment* ones (247).

<figure><img src="image_4.png" /><figcaption>Figure 1: Count of occurrences of inference types labels and NLI labels in the test set.</figcaption></figure>

We also provide these annotations.

## 3.2. Inference Types: Definitions

We define super-classes of inference types that comprise more fine-grained types. Fig. 3 illustrates the complete schema.

**Task-related expression (TRE):** any expression that refers to clinical trials or the structure of a clinical trial report. For instance, the different *cohorts*, *arms*, or *groups*.

E.g., **Statement (S):** *Patients diagnosed with breast cancer may be eligible for all study groups in the primary trial and the secondary trial.*

**Premise (P):** *Inclusion Criteria: Healthy Participants (Part 1 only) ... HER2-Positive Females (Parts 1 and 2) ... Exclusion Criteria: Healthy Participants (Part 1 only) ... HER2-Positive Females (Parts 1 and 2).* ⇒ part 1 and part 2 are the study groups, while inclusion and exclusion criteria are determining the patient's eligibility.

**Logical (super-class):** connectives and quantifiers, as specified below.

**Conjunction (Conj):** terms in the statement expressing a logical AND.

E.g., **S: The primary trial intervention section requires surgical and imaging procedures.**

**Disjunction (Disj):** terms in the statement expressing a logical OR.

E.g., **S: Aes were not recorded for the primary trial or the secondary trial.**

**Negation (Neg):** a term in the statement expressing that some assertion is not true.

E.g., **S: Aes were not recorded for the primary trial or the secondary trial.**

**Contrast:** some terms in the statement expressing a contrast between two clauses, using words like *but*, *whereas*, *however*, etc.

E.g., **S: Children are not eligible for the primary trial however they are not explicitly excluded from the secondary trial.**

<footnote n="2">²Annotation guide</footnote>
<figure><img src="image_5.png" /><figcaption>Figure 2: Correlation matrix of the inference type labels.</figcaption></figure>

<figure><img src="image_6.png" /><figcaption>Figure 3: Our categorization of inference types in NLI4CT.</figcaption></figure>

**Universal quantifier (UQ):** A term in the statement asserts that a condition applies to all elements in a specified set (∀ symbol).

E.g., **S:** *Patients diagnosed with breast cancer may be eligible for all study groups in the primary trial and the secondary trial.*

**Existential quantifier (EQ):** a term in the statement asserts that a condition applies to at least one element in a specified set (∃ symbol).

E.g., **S:** *In total there are less cases of anemia in the primary trial than in the secondary trial.*

**Numerical (super-class):** fine-grained numerical inference types that include arithmetic operations, conversions, comparisons, and manipulation of numbers and quantities.

**Numerical comparison (NC):** when two numbers or quantities need to be compared and checked whether the number mentioned in the statement matches, is greater than, etc., the one in the premise.

E.g., **S:** *over 15% of patients in the primary trial and the secondary trial suffered from infections during the study period.*

**P:** Primary trial: *Infection* * 1/32 (3.13%). Secondary trial: *Infections and infestations - Other, unspecified 1/15 (6.67%) ... Infections and infestations - Other, unspecified 0/5 (0.00%).*

**Numerical arithmetic (NA):** when an arithmetic operation (addition, subtraction, multiplication, division) is needed to solve the inference relation.

E.g., **S:** *All Patients receiving the placebo intervention in the primary trial experienced emesis.*

**P:** *Outcome Measurement: Number of Emesis Free Participants During the Study Period. ... Overall Number of Participants Analyzed 20. Measure Type: Number. Unit of Measure: Participants 5. ⇒ 5/20 = 25% ≠ 100%, which means that not all participants experienced emesis.*

**Unit conversion (UC):** a conversion from one unit to another is required to solve the inference relation.

E.g., **S:** *A Patient that has a primary tumour with a diameter of 33mm measured by clinical examination and echography, would be eligible for both the primary trial and the secondary trial.*

**P:** *Primary tumour greater than 2 cm diameter... ⇒ convert the millimeters into centimeters to be able to compare the 2 measures.*

**Fuzzy quantity (FQ):** a quantity subject to several interpretations is present in the statement, and its interpretation is needed to solve the inference relation, using words like *almost*, *several*, etc.

E.g., **S:** *Several recorded Aes in the primary trial occurred to cohort 1 patients.*

**Biomedical (super-class):** biomedical terms and knowledge that are needed to solve the inference relation.
**Ancestor-descendant relation (ADR):** terms that are either children or parents of the considered term in the premise.

E.g., **S:** There was at least 1 recorded gastro-intestinal adverse event in the primary trial.

**P:** Vomiting 0/149 (0.00%). ⇒ vomiting is a child of gastro-intestinal adverse event.

**Biomedical acronym (BA):** acronyms that shorten a biomedical term or expression. This acronym should be listed in one of the terminologies of the UMLS.

E.g., **S:** Women suffering from both claustrophobia and IBS or not eligible for either the primary trial or the secondary trial. ⇒ IBS = Irritable Bowel Syndrome.

**Biomedical notion definition (BD):** instances that require knowing the definition of a medical term and what properties it implies, and then to compare these properties with the information in the premise.

E.g., **S:** Morbidly obese patients are eligible for the primary trial.

**P:** Have a BMI of 25 kg/m2 or greater. ⇒ Morbidly obese patients have a BMI greater than 40 kg/m².

**Biomedical synonym (BS):** two biomedical terms, one in the statement and one in the premise, that are equivalent.

E.g., **S:** Both interventions in the primary trial include the same dose of Paraplatin.

**P:** INTERVENTION 2: Arm 2 Taxotere/carboplatin/herceptin. ⇒ Paraplatin is the brand name of Carboplatin.

**Generic knowledge (GK):** as opposed to domain-specific knowledge, notions that do not fall into the previous categories. In our case, these are often temporal words (months, hours) or concepts such as age, gender, ethnicity, nationality, etc.

E.g., **S:** There are no conditions on mental health, bodyweight, age, Karnofsky/ECOG score or previous treatments that need to be met in order to be eligible for the primary trial.

**P:** Inclusion Criteria: Women 18 years...

**Typo/error (TE):** we annotate a typo or error in the statement or the premise that would lead to a major misunderstanding of them and have an impact on the inference relation resolution of statement and premise(s).

E.g., **S:** The *the primary trial intervention section* dose not describe the method of administration, dosage or cycle. ⇒ *The* appears twice and *does* is replaced by *dose*.

## Comparison to previous annotation schemes

The original annotations provided with NLI4CT, only separates the examples involving a numerical inference and does not provide more precise categories.

³ We computed the overlap of instances labeled as *Numerical* in NLI4CT with numerical instances using our definition, which shows that we share 82% of instances labeled as *Numerical*. This suggests that a similar definition was used in the two studies. Our *Numerical* super-class aligns with those of ANLI (Nie et al., 2020) and Williams et al. (2022), and shares the *Mathematical* aspect of TaxiNLI's *Logical*. Our *Logical* largely encompasses TaxiNLI's *Logical*, which corresponds to ANLI's *Standard*, and Williams et al. (2022)'s *Base* for conjunction and negation. *Generic knowledge* corresponds to some aspects of ANLI's and Williams et al. (2022)'s *Reasoning*, and TaxiNLI's *Knowledge*. *Typo/error* maps to Williams et al. (2022)'s *Imperfections*. *Biomedical* does not really map to any categories in other schemes and can be considered original since all the previous schemes are dealing with general-domain data.

## 3.3. Prompting Large Language Models

We select open-source LLMs, highest ranked in SemEval 2023 (Jullien et al., 2023b) and 2024 (Jullien et al., 2024). We evaluate the following models: Flan-T5-xl and xxl (Chung et al., 2024), Mistral-7B-Instruct-v0.1 (Jiang et al., 2023), Mistral-8x7B-Instruct-v0.1 (Jiang et al., 2024), Llama-3.2-8B-Instruct (Dubey et al., 2024), Qwen2.5-7B and 14B-Instruct (Yang et al., 2024), all pretrained for the general domain. In addition, we use MMed-Llama-3-8B-EnIns (Qiu et al., 2024), a Llama3 model finetuned on the medical domain.

We use the same template as Kanakarajan and Sankarasubbu (2023) and added the mention "Answer only with:" to better constrain models to output the desired labels:

*{Premise} Question: Does this imply that {Statement}? Answer only with: {options}, with options being Entailment and Contradiction.*

We performed in a zero-shot setting, and used a temperature of 0.7, a top_p of 1.0, and top_k of 0. We set the maximum number of generated tokens to 10 and parse the produced answers using regular expressions. Accuracy is used to report the model's performance.

# 4. Results and Discussion

## 4.1. Overall Performance

In Tab. 2, we report the mean global accuracy (in column *All types*) to predict *Entailment* or *Contradiction* for all the instances of the test set of NLI4CT, using the template described in Sec. 3.3. All the

<footnote n="3">³Even though the paper talks about biomedical, common sense and numerical reasoning, the annotations only include numerical labels.</footnote>
experiments are run three times, each with a different random seed (42, 55, and 3354). Qwen-14B achieves the best results with an accuracy of 0.73; on the other hand, MMed-Llama performs the worst with an accuracy of 0.55, which is close to random.

## 4.2. Performance per Inference Type

For each inference type, we compute the mean accuracy on the 3 runs. We define 2 subsets: the *i* subset, where we compute the accuracy only on instances labeled with inference type *i*, and the *ī* set, where we compute the accuracy on all the instances that are not labeled with *i*.

To test whether the difference of performance between *i* and *ī* is statistically significant, we perform a Chi-square (χ²) test (Agresti, 2013) with a p-value threshold of 0.05. We define χ²-*type*, where, for each model, the χ² is computed on one *i* and *ī* inference type subsets (see Eq. 1). The null hypothesis is: “When inference type *i* is present, the model performs as well as when the inference type *i* is not present.”

$$ \chi_{type}^2 = \sum_{k \in \{i, i\}} \frac{(O_k - E_k)^2}{E_k} \quad (1) $$

Where $O_k$ is *k*'s observed frequency and $E_k$, *k*'s expected frequency. For each of the three runs of a system, we compute its mean χ²-*type* and the associated p-value. In all cases, the p-value of the three runs is on the same side of the threshold.

Tab. 2 reports these results, where highlighted cells show the pairs of results with significant differences. Globally, not all models are sensitive to the different types of inference involved. Best performing models (with a minimum of 17 points above randomness) are sensitive to the different types of inference. Even across different versions of the same type of model (for instance, Flan-xl versus Flan-xxl), we do not necessarily observe a sensitivity. However, both *Logical* and *Numerical* are challenging classes.

For Flan-xl, the overall model's performance is impacted by *Numerical*, especially *Numerical comparison*, where $NC = 0.63$ and $NC̄ = 0.74$, and *Unit conversion*, where $UC = 0.46$ and $UC̄ = 0.68$. Mixtral is negatively impacted by *Unit conversion* for *Numerical*, $UC = 0.46$ and $UC̄ = 0.68$, but also by *Negation* with $Neg = 0.60$ and $Neḡ = 0.70$, and *Disjunction* with $Disj = 0.58$ and $Disj̄ = 0.69$, for *Logical*. Qwen-7 is negatively impacted by *Negation* with $Neg = 0.61$ and $Neḡ = 0.72$ and *Contrast* with $Contrast = 0.60$ and $Contrast̄ = 0.71$. Qwen-14 is also impacted by *Negation* with $Neg = 0.58$ and $Neḡ = 0.77$ and *Disjunction* with $Disj = 0.62$ and $Disj̄ = 0.75$. While Jullien et al. (2023a) also had similar observations, where models struggled more with numerical inference than other types of

inference, we found that *Logical* also has a significant impact on models' performance. In addition, as displayed in Fig. 2, *Negation* and *Disjunction* are highly positively correlated, which suggests that the presence of the two together makes it even harder for the models to deal with. In contrast, the other challenging labels are not correlated with each other.

# 5. Error Analysis

In this section, we perform a more in-depth analysis of our results. For better readability, we focus on the 2 best performing models, the worst performing one, and Flan-xl that has a Seq2Seq architecture, unlike the others.

## 5.1. Are LLMs able to understand the CTR structure?

Understanding the structure of a CTR is both essential and challenging in order to solve the NLI task. Indeed, the model needs to understand the different cohorts, groups, or arms, as well as their corresponding results, interventions, eligibility criteria, or adverse events. This is the first step to understand the statement and to consider the right pieces of evidence to solve the inference relation.

All test set instances were annotated with *Task related expressions*, meaning that understanding the structure of CTR and that the statement and the premise are related is a mandatory step of reasoning. Most often, other reasoning types co-exist with this label. 12 instances of the dataset were only labeled with *Task related expression* and no other reasoning types. Tab. 3 displays the performance of different models on this label.

The results show that Flan-T5-xl and both Qwen 7 and 14 seem to be able to deal with CTR structure quite well since their performance on *TR-only* is greater than the performance on all types of inference. On the other hand, Mmed-Llama exhibits a poor performance, which suggests that the model does not understand the CTR structure and can explain why it struggles to solve examples involving multiple inference types.

## 5.2. Is the number of inference types involved having an impact on performance?

We hypothesize that having a higher number of different types of inference would lead to a decrease in performance, due to the multiple types of knowledge that the model needs to put in contrast.

Fig. 4 shows the mean accuracy per number of inference types involved in an example.
<table><caption>Table 2: Mean accuracy per super-classes of inference type on the i and i&#773; subsets, in decreasing order of All-types accuracy. GK = *Generic knowledge*, Num = *Numerical*, Bio = *Biomedical*, Log = *Logical*, TE = *Typo/Error*. *Task-related expression* is not reported since it appears in every example of the dataset, which means its performance is equivalent to All types. Highlighted cells show pairs of results with statistically significant differences between the presence and absence of the inference type. Standard deviations are all less than or equal to 0.01, so we do not report them in the table.</caption><thead><tr><th>Model</th><th>Log</th><th>Log&#773;</th><th>Num</th><th>Num&#773;</th><th>Bio</th><th>Bio&#773;</th><th>GK</th><th>GK&#773;</th><th>TE</th><th>TE&#773;</th><th>All types</th></tr></thead><tbody><tr><td>MMed-Llama-3</td><td>0.56</td><td>0.55</td><td>0.51</td><td>0.55</td><td>0.56</td><td>0.55</td><td>0.45</td><td>0.56</td><td>0.53</td><td>0.55</td><td>0.55</td></tr><tr><td>Llama-3</td><td>0.60</td><td>0.55</td><td>0.55</td><td>0.56</td><td>0.55</td><td>0.56</td><td>0.56</td><td>0.56</td><td>0.69</td><td>0.55</td><td>0.56</td></tr><tr><td>Mistral</td><td>0.58</td><td>0.59</td><td>0.59</td><td>0.58</td><td>0.62</td><td>0.58</td><td>0.60</td><td>0.59</td><td>0.64</td><td>0.58</td><td>0.59</td></tr><tr><td>Flan-T5-xl</td><td>0.68</td><td>0.67</td><td>0.55</td><td>0.70</td><td>0.68</td><td>0.67</td><td>0.67</td><td>0.67</td><td>0.67</td><td>0.66</td><td>0.67</td></tr><tr><td>Flan-T5-xxl</td><td>0.69</td><td>0.66</td><td>0.63</td><td>0.68</td><td>0.68</td><td>0.67</td><td>0.61</td><td>0.67</td><td>0.73</td><td>0.67</td><td>0.67</td></tr><tr><td>Mixtral</td><td>0.65</td><td>0.68</td><td>0.64</td><td>0.68</td><td>0.65</td><td>0.68</td><td>0.61</td><td>0.68</td><td>0.62</td><td>0.68</td><td>0.67</td></tr><tr><td>Qwen-7B</td><td>0.67</td><td>0.70</td><td>0.70</td><td>0.69</td><td>0.70</td><td>0.69</td><td>0.68</td><td>0.69</td><td>0.67</td><td>0.69</td><td><u>0.69</u></td></tr><tr><td>Qwen-14B</td><td>0.70</td><td>0.73</td><td>0.73</td><td>0.73</td><td>0.73</td><td>0.73</td><td>0.62</td><td>0.73</td><td>0.68</td><td>0.73</td><td><b>0.73</b></td></tr></tbody></table>

<table><caption>Table 3: Mean accuracies on three runs, on examples labeled only as *Task-related expression* (TRE).</caption><thead><tr><th>Model</th><th>TRE-only</th></tr></thead><tbody><tr><td>MMed-Llama-3</td><td>0.19</td></tr><tr><td>Flan-T5-xl</td><td>0.75</td></tr><tr><td>Qwen-7B</td><td>0.86</td></tr><tr><td>Qwen-14B</td><td>0.81</td></tr></tbody></table>

<figure><img src="image_7.png" /><figcaption>Figure 4: Evolution of the mean accuracy on the three runs in function of the number of inference types involved in the considered example.</figcaption></figure>

As mentioned in Sec. 5.1, when only one instance is present, this is necessarily *Task-related expression*. All models except Mmed-Llama achieve one of their highest performances in that case. When introducing two types of inference, the performances of models tend to slightly increase or remain stable, except for Qwen-7B, whose performance slightly decreases. On the contrary, the performance of MMed-Llama-3 almost triples. Generally, adding more inference types simultaneously is more challenging. Surprisingly, Qwen models and Mixtral reach a peak of performance when seven types of inference are present. However, all models fall short when facing eight types of infer-

ence. Note, though, that only 13 (resp. 4) examples in our dataset are labeled using 7 (resp. 8) inference types, whereas 67 examples populate the other performance peak where 2 inference types are involved. Qwen and Mixtral were all struggling with *Negation*. For examples involving 2 inference types, *Negation* is only present in 4% of the examples, meanwhile for examples involving 7 types *Negation* is present for 77% of the examples. We hypothesize that in the case of 7 inferences, the presence of the other types of inference compensates the difficulty of *Negation*, while in the case where 2 inference types are present, there are no other types to compensate, but since *Negation* represents only 4% of the instances, the model remains with a high performance.

## 5.3. Is Comparison harder than Single?

*Comparison* examples need to confront two CTRs to a given statement. One would expect them to be harder because the model needs to deal with two premises and identify more pieces of evidence compared with a *Single* example. Fig. 5 shows the accuracy of the models for *Comparison* and *Single* examples in function of the inference type involved.

For the 4 models, *Single* remains easier than *Comparison*. However, depending on the type of inference involved, this observation varies: Qwen-7B achieves a higher performance in *comparison* setting for *Existential quantifier* and *Fuzzy quantity*, while Flan-xl and Qwen-14B achieve higher performance for *Typo/error*, and MMed-Llama-3 for *Ancestor-descendant relation*. Some inference types in a *Comparison* setting are challenging for all models: *Biomedical synonym*, *Biomedical notion definition*, and *Generic knowledge*. These types of inferences particularly require comparing several pieces of text, e.g., when comparing synonyms in the first and second CTR premises with the state-
<figure><img src="image_8.png" /><figcaption>(a) Flan-xl</figcaption></figure>

<figure><img src="image_9.png" /><figcaption>(b) MMed-Llama-3</figcaption></figure>

<figure><img src="image_10.png" /><figcaption>(c) Qwen-7B</figcaption></figure>

<figure><img src="image_11.png" /><figcaption>(d) Qwen-14B</figcaption></figure>

<caption>Figure 5: Accuracy of the different models for Com-parison and Single examples in function of the in-ference type involved.</caption>

ment.

## 5.4. Adverse Events vs Results vs Intervention vs Eligibility

In each example, the premise is composed of one whole section, either *Adverse Events*, *Results*, *Intervention*, or *Eligibility*. Each section has its own structure and involves different types of knowledge. Fig. 6 displays the mean accuracy of each model per CTR section.

<figure><img src="image_12.png" /><figcaption>Figure 6: Mean accuracy in function of the section considered as the premise.</figcaption></figure>

We also compute the top-3 most frequent infer-ence type labels present for each section to know what kind of knowledge is most frequently needed to solve an inference involving a given section. While *Eligibility* and *Intervention* are mostly labeled with inference types belonging to the *Logical* super-class, *Adverse Event* is as much labeled using *Nu-merical* and *Logical* inference types, and *Results* is strongly dominated by *Numerical* inference types.

For Qwen models, the *Eligibility* section is the most challenging with up to 6 points of difference with the other sections. Flan-T5-xl has a gap of up to 11 points between the *Adverse Events* and *Results* sections. MMed-Llama-3 achieves almost random performance for all sections, except for *Adverse Events* with up to 13 more points. We note that the sections on which LLMs struggle are the ones with a higher presence of the inference type whose presence leads to a significant decrease in performance.

# 6. Conclusion

In this study, we proposed a definition of inference types along with new annotations on the NLI4CT dataset for natural language inference on clinical trials. We defined 17 different types of inferences present in NLI4CT, clustered in 6 super-classes: *Task related expression*, *Logical*, *Biomedical*, *Nu-merical*, *Generic knowledge*, and *Typo/Error*. We believe that these definitions and methodology could also be used for general-domain or other domain-specific applications.

We investigated the influence of each inference type on the performance of several open-source LLMs and found that not all models are sensitive to the types of inference involved. However, passed a threshold of performance, most of the models exhibit a sensitivity. Generally, there is a signifi-cant drop in performance on logical and numerical inference types. We found that Flan-T5-xl is sensi-tive to *Unit conversion* and *Numerical Comparison*, *Mixtral*, Qwen-7B and 14B are all sensitive to *Nega-
tion but Qwen-7B is also sensitive to *Contrast* and Qwen-14B to *Disjunction*. Mixtral is also sensitive to *Unit conversion*. For future work, we plan to run the same experiments in a few-shot setting or by using Chain-Of-Thought to see whether such approaches would improve the results. We also plan on looking into LLMs' weak points by analyzing the natural language explanations associated with the predicted labels and seeing if these explanations correlate with our observations.

## 7. Limitations

The disagreement between the annotators highlights the complexity of the annotation task. Sometimes, there is an overlap between inference types such as *Ancestor-descendant relation* and *Biomedical synonym*, or *Generic Knowledge* and *Unit conversion* when it comes to dealing with months, hours, etc., which led to many discussions during the annotation process. As stated by Pavlick and Kwiatkowski (2019), these disagreements can reflect the full distribution of plausible human judgments. To provide a better understanding of the possible annotations produced during our process, we also release the 50 instances annotated by the three annotators and the corresponding justification for each instance.

## 8. Ethical Considerations

The NLI4CT task uses clinical data extracted and processed from clinicaltrials.gov. This resource is freely available, provided by the National Library of Medicine, and is an official U.S. Department of Health and Human Services website.

All annotators are NLP researchers, authors of this paper, and paid by their own institutions. They gave consent to annotate this dataset as part of their research activities.

## 9. Acknowledgements

## 10. Bibliographical References

- Alan Agresti. 2013. *Categorical data analysis*. John Wiley & Sons.

- Hyung Won Chung et al. 2024. Scaling instruction-finetuned language models. *Journal of Machine Learning Research*, 25(70):1–53.

- Ido Dagan, Oren Glickman, and Bernardo Magnini. 2006. The pascal recognising textual entailment challenge. In *Machine Learning Challenges. Evaluating Predictive Uncertainty*, Visual

- Object Classification, and Recognising Tectual Entailment, pages 177–190, Berlin, Heidelberg. Springer Berlin Heidelberg.

- Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In *Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)*, pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.

- Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurélien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Rozière, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dihan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Graeme Nail, Grégoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel M. Kloumann, Ishan Misra, Ivan Evtimov, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, and et al. 2024. The Llama 3 herd of models. *CoRR*, abs/2407.21783.

- Clémentine Fourrier, Nathan Habib, Alina Lozovskaya, Konrad Szafer, and Thomas Wolf. 2024. Open llm leaderboard v2. https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard.
Albert Q. Jiang et al. 2023. Mistral 7b. ArXiv, abs/2310.06825.

Albert Q. Jiang et al. 2024. Mixtral of experts. ArXiv, abs/2401.04088.

Pratik Joshi, Somak Aditya, Aalok Sathe, and Monojit Choudhury. 2020. TaxiNLI: Taking a ride up the NLU hill. In *Proceedings of the 24th Conference on Computational Natural Language Learning*, pages 41–55, Online. Association for Computational Linguistics.

Mael Jullien, Marco Valentino, and André Freitas. 2024. SemEval-2024 task 2: Safe biomedical natural language inference for clinical trials. In *Proceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024)*, pages 1947–1962, Mexico City, Mexico. Association for Computational Linguistics.

Mael Jullien, Marco Valentino, Hannah Frost, Paul O'Regan, Dónal Landers, and Andre Freitas. 2023a. NLI4CT: Multi-evidence natural language inference for clinical trial reports. In *Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing*, pages 16745–16764, Singapore. Association for Computational Linguistics.

Maël Jullien, Marco Valentino, Hannah Frost, Paul O'regan, Donal Landers, and André Freitas. 2023b. SemEval-2023 task 7: Multi-evidence natural language inference for clinical trial data. In *Proceedings of the 17th International Workshop on Semantic Evaluation (SemEval-2023)*, pages 2216–2226, Toronto, Canada. Association for Computational Linguistics.

Kamal Raj Kanakarajan and Malaikannan Sankara-subbu. 2023. Saama AI research at SemEval-2023 task 7: Exploring the capabilities of flan-t5 for multi-evidence natural language inference in clinical trial data. In *SemEval-2023*, pages 995–1003, Toronto, Canada.

Kyle Mahowald, Anna A. Ivanova, Idan A. Blank, Nancy Kanwisher, Joshua B. Tenenbaum, and Evelina Fedorenko. 2024. Dissociating language and thought in large language models. *Trends in Cognitive Sciences*, 28(6):517–540.

Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020. Adversarial NLI: A new benchmark for natural language understanding. In *Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics*, pages 4885–4901, Online. Association for Computational Linguistics.

Ellie Pavlick and Tom Kwiatkowski. 2019. Inherent disagreements in human textual inferences.

*Transactions of the Association for Computational Linguistics*, 7:677–694.

Pengcheng Qiu, Chaoyi Wu, Xiaoman Zhang, Weixiong Lin, Haicheng Wang, Ya Zhang, Yanfeng Wang, and Weidi Xie. 2024. *Towards building multilingual language model for medicine*. *Nature Communications*, 15(1).

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. *Attention is all you need*. In *Advances in Neural Information Processing Systems*, volume 30. Curran Associates, Inc.

Adina Williams, Tristan Thrush, and Douwe Kiela. 2022. ANLIzing the adversarial natural language inference dataset. In *Proceedings of the Society for Computation in Linguistics 2022*, pages 23–54, online. Association for Computational Linguistics.

Liyun Yan, Danni E, Mei Gan, Cyril Grouin, and Mathieu Valette. 2020. Inference annotation of a Chinese corpus for opinion mining. In *Proceedings of the Twelfth Language Resources and Evaluation Conference*, pages 4991–4999, Marseille, France. European Language Resources Association.

An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. 2024. *Qwen2.5 technical report*. arXiv preprint arXiv:2412.15115.