You are an expert in Deep Learning and Machine Learning. As an Academic Peer Review Assistant, your role is to provide a thorough and constructive review of submitted academic papers. When a user presents a paper for review, assess it across several key dimensions to ensure scholarly rigor and clarity. Your evaluation should cover:

•              Content Accuracy and Relevance: Analyze whether the information presented is accurate, up-to-date, and relevant to the stated research questions.

•              Argumentation and Analysis: Assess the strength and coherence of the arguments. Are the claims well-supported by evidence? Is the analysis thorough and critical?

•              Methodology: Evaluate the appropriateness and execution of the research methodology. Is it suitable for the research question? Are the methods clearly explained and justified?

•              Organization and Structure: Consider the logical flow of the paper. Is it well-organized, allowing readers to easily follow the progression of ideas?

•              Writing Quality and Clarity: Review the paper for clarity of language, style, and grammar. Is the writing clear, concise, and appropriately academic?

•              References and Citations: Check for the completeness and accuracy of citations and references. Are all sources credible and properly attributed?

Provide specific feedback for each category, citing examples from the paper to support your evaluations. Offer constructive criticism and suggestions for improvement to help the author enhance the paper. Summarize the overall strengths and weaknesses, and recommend whether the paper should be accepted, revised, or rejected based on your assessment.


Review this paper:
# A Reproducibility Study on

# Consistent LLM Reasoning for

# Natural Language Inference over Clinical Trials

```
Anonymous Author(s)
```
```
Anonymous Institute(s)
```
```
Abstract. With the rapid expansion of AI in healthcare, ensuring that
language models can accurately and consistently reason within the medi-
cal domain is essential for enhancing clinical decision-making. Consistent
reasoning is particularly challenging, as once a model outputs a judgment
for a given medical statement, it should retain that judgment when faced
with another statement that is only syntactically altered. Also, when the
other statement reflects a semantic shift, the model should adjust its
judgment accordingly. In this paper, we describe the process of repro-
ducing state-of-the-art methods for safe biomedical Natural Language
Inference for Clinical Trials (NLI4CT), emphasizing model vulnerability
to small input variations and the inherent complexity of clinical trial
reports (CTRs). We specifically evaluate the reasoning capabilities of
Large Language Models (LLMs) in the SemEval-2024 NLI4CT dataset,
thus considering a task that focused on robustness, and which serves as
a good proxy for real-world applications. To improve thoroughness, we
extend this study and explore a broader set of techniques, establishing
baseline scores across several widely used models. We conclude with an
analysis of the results, highlighting key insights and empirical lessons
that contribute to future research in this domain.
```
```
Keywords: Natural Language Inference·Reasoning with LLMs·
Clinical Trials·SemEval-2024·Reproducibility.
```
## 1 Introduction

Clinical Trials (CTs) [15,39] are research studies that intend to safely introduce
novel medical treatments [2]. These studies are documented in Clinical Trial Re-
ports (CTRs), which provide concise descriptions for the objectives of a trial,
its methodologies, eligibility criteria, and outcomes. Clinical experts can for in-
stance rely on CTRs to evaluate the suitability of administering new treatments
to patients, which is a complex reasoning task.
Large Language Models (LLMs) offer significant potential in assisting clinical
reasoning e.g. by, interpreting and deriving conclusions from CTRs. Nonethe-
less, several critical challenges limit their reliability, including data contamina-
tion [24,7,11], fairness concerns [32,22], generation of hallucinations [40,1], or
lack of explainability [6,41]. Most importantly for this work, the nonexistence


2 Anonymous Author(s)

```
Section ID: Interventions
Primary Clinical Trial:
INTERVENTION 1: PF-06647020 0.2 mg/kg (08Q3W Regimen); Participants enrolled in
the dose escalation phase received PF-06647020 at 0.2 mg/kg on Day 1 of each 21-day
cycle ...
INTERVENTION 2: PF-06647020 0.5 mg/kg (Q3W Regimen); Participants enrolled in the
dose escalation phase received PF-06647020 at 0.5 mg/kg on Day 1 of each 21-day
cycle ...
Secondary Clinical Trial:
INTERVENTION 1: ALT-801 0.015 mg/kg/Dose;
INTERVENTION 2: ALT-801 0.040 mg/kg/Dose
Statement: The drug dosage for the primary clinical trial participants is lower in
comparison to that of the secondary clinical trial participants.
```
```
Fig. 1. Example of an entailment/contradiction query in the clinical domain.
```
of robust evaluation metrics specifically designed to assess a model’s reasoning
ability, combined with limited comprehensive benchmarks [28,8,27] for CTRs,
poses significant concerns in ensuring safe and reliable systems.
To enhance evaluation methodologies for clinical Natural Language Infer-
ence (NLI), SemEval-2024 [25] organized the Safe Biomedical Natural Language
Inference for Clinical Trials (NLI4CT) task [16]. This task involved the classifica-
tion of statements related to CTRs, establishing a framework for more effective
assessment of LLMs in the context of clinical reasoning. More precisely, the pri-
mary objective was to support the development of systems capable of predicting
whether a given statement can be entailed or contradicted by the information
contained within one or two CTRs, in a consistent and coherent manner.
Considering that the same CTR can be associated to different variations
of a base statement, systems participating in the NLI4CT task were required
to demonstrate consistent responses to these modifications. Desired properties
include accuracy and robustness in the face of redundant perturbations to the
statements, and also alignment between semantic changes in the input and the
system’s output. The idea is to ensure that systems exhibit known and expected
error modes, this way reinforcing reliability and interpretability.
As the main contribution of this paper, we reproduce and publish the imple-
mentations of state-of-the-art NLI methods in the clinical domain. This effort
offers as a rigorous and reproducible empirical study designed to advance clinical
reasoning research. In addition to reproducing previous methods, we conducted a
series of experiments that extend current methodologies, yielding notably strong
results that not only reaffirm existing baseline performances, but also provide
new insights into their efficacy.

## 2 Related Work

This section contextualizes the available resources for clinical NLI, and provides a
summarized overview of the main techniques that performed well in the SemEval-
2024 NLI4CT challenge.


```
Consistent LLM Reasoning for NLI over Clinical Trials 3
```
Julien et al. [17] provided an exposition on benchmark datasets available in
the domain of clinical trials: the data from the **TREC 2021 Clinical Track** [29,27],
focused on associating synthetic patient cases to CTRs by retrieving the most rel-
evant results to each patient, evaluated by nDCG and precision; The **MEDNLI** [28]
dataset, that focuses entailment relations between patient medical records and
annotated sentences about those records, evaluating accuracy; and **Evidence
Inference 2.0** [8], in which systems have to perform Question Answering (QA)
with the goal of predicting the relationship between given CTR data and a treat-
ment, evaluated by macro F1, AUC, and an evidence token metric. The NLI4CT
dataset [17] distinguishes itself from the aforementioned datasets by utilizing
full CTR sections in its queries, by containing examples that intend to prove
a model’s ability to perform numerical inference and common-sense reasoning,
and by focusing on ensuring robustness to perturbations on statements.

NLI4CT is a particularly important benchmark dataset in the clinical do-
main because it specifically targets the LLMs reasoning capacities. The following
teams submitted interesting approaches and achieved top scores in the SemEval-
2024 NLI4CT challenge [16]: e.g. Gema et al. [10] (Edinburgh Clinical NLP);
Lee et al. [21] (NYCU-NLP); Liu and Thoma [23] (FZI-WIM); and Guimarães
et al. [12] (Lisbon Computational Linguists). A common starting point was to
assess the **zero-shot** performance of several LLMs, i.e., ot providing examples
or a refined prompting strategy. The standout result from this process was ob-
tained by utilizing GPT-4 [26], which outperformed all other approaches, whilst
other experiments served as a baseline result to ablate the performance gains
from training procedures or advanced prompting techniques.

Previous studies demonstrated the effects of **prompting** [5] on the capability
that a LLM has to follow instructions, exposing emergent abilities from the
pre-training process [4,37]. To this effect, teams explored two main prompting
strategies: **Chain-of-Thought (CoT)** reasoning [38], instructing the model
to explain its sequential inference process with specific key phrases (e.g., “ _let’s
think step by step_ ” or “ _verify it step by step_ ”) [19]; and **In-Context Learning
(ICL)** [34], where examples of one or more successful answers to CTR-Statement
pairs were added to the prompt. Finally, **Self-Consistency** was also tested in
conjunction with the aforementioned techniques [35], taking advantage of the
ability to generate non-deterministic outputs (e.g., using sampling with a given
temperature, top- _p_ and top- _k_ parameters) to explore different reasoning chains,
and choosing the final label by majority voting.

Advancing to techniques that require training, all the aforementioned sub-
missions to the NLI4CT task explored using **instruction fine-tuning** [36] on
open-source LLMs to improve their reasoning abilities, testing mostly models
based on Mistral-7B [14] and LLaMa-2 [31]. To increase the amount and qual-
ity of fine-tuning data, some authors **augmented the available data** , using
Chat-GPT v3.5 [18], GPT-4, or Mistral-7B to either explain training examples
(e.g., using CoT prompts [13]) or to create examples from the original statements
(e.g., paraphrasing, contradicting or elaborating on the original statements).


4 Anonymous Author(s)

```
Original Statement: Only 2 patients in the primary trial did not have Recurrence-free
Survival (Label:Entailment)
(a) Variation–Paraphrase: Only two individuals in the primary clinical trial were not
recurrence-free survivors (Label:Entailment)
(b) Variation–Contradiction: Adverse events were a common occurrence among partic-
ipants in the primary trial (Label:Contradiction)
(c) Variation–Text Append: Aggravated malignant neoplasm is a malignant neoplasm
that shows clinical and/or pathologic progression. Only 2 patients in the primary trial did
not have Recurrence-free Survival. (Label:Entailment)
(e) Variation–Num. Contradiction: 3% of patients in the primary trial did not experi-
ence any adverse events. (Label:Contradiction)
```
```
Fig. 2. Examples of variations on a clinical statement.
```
## 3 Aspects Involved on Consistent Reasoning Methods

This section highlights the key challenges of the task at hand, including the
complexity of consistent reasoning and the nuanced nature of CTRs. It then
introduces the state-of-the-art methods that were implemented for this study,
detailing the rationale behind each model choice.

**3.1 Consistent Reasoning about Clinical Trials**

Building on the previously established challenges, the core concept behind
consistent reasoning is to design systems that arrive at their conclusions based
on valid reasoning processes, even if their final responses are incorrect. This is
crucial because, without insight into the specific factors driving a system’s judg-
ment, it becomes difficult to assess its true reliability. Therefore, it is essential to
develop evaluation methodologies that provide a deeper understanding of how
dependable the systems are.
Figure 2 illustrates this process with a practical example. Consider the origi-
nal statement “ _only two patients in the primary trial did not have recurrence-free
survival_ ,” analyzed within the context of a CTR. The correct classification for
the example Statement-CTR pair is “ _entailment_ ”. If we were to proceed with-
out further assessment, we would have little assurance that the system truly
understood and analyzed the example, or if it merely reached the conclusion
through reasoning shortcuts. To address this uncertainty, we introduce variations
of the statement to test the model’s robustness: (a) simple paraphrasing exam-
ines whether the system can maintain its judgment despite syntactic changes; (b)
contradiction tests its ability to adjust the label in response to semantic shifts;
(c) appending text introduces irrelevant noise; (d) numerical paraphrasing alters
quantities whilst keeping meaning (e.g., changing “ _2/67 patients_ ” to “ _3%_ ”); and
(e) numerical contradiction changes quantities and their meaning. Having estab-
lished the importance of consistent reasoning and the need for robust evaluation
methodologies, we now turn our attention to method selection.


```
Consistent LLM Reasoning for NLI over Clinical Trials 5
```
**3.2 Baseline Methods based on LLMs**

After reviewing the average scores of the twenty-five systems that were evalu-
ated in the SemEval-2024 NLI4CT task [16], we focused on the success of the
top methods discussed in Section 2. Before reproducing their work, we elected
to establish a comprehensive and unbiased analysis of **zero-shot** results, as a
baseline for all other techniques. We opted for a selection of **LLMs** that allowed
us to reproduce (not replicate) the top submissions, namely the following:

1. **Mistral-7B** based models: **Mistral-7B-instruct-v0.2**^1 , fine-tuned with in-
    struction prompts; **BioMistral-7B** [20]^2 , pre-trained with medical data; and
    **MistralLite-7B**^3 , fine-tuned with longer text sequences.
2. **LLaMa3.1-8B-instruct** [9]^4 , a recent version of the LLaMa herd of models;
3. **Gemini** [30], a commercial black-box model chosen to emulate GPT-4.

As we focused on **zero-shot** performance only, we did not use ICL, CoT, or
self-consistency in our baselines. Each model was **prompted** with both a **short**
and a **long prompt**^5 , where the short prompt provided minimal context and
the long prompt detailed the structure of a CTR. The prompts were adapted to
suit each model’s templating requirements (e.g., correct use of special tokens).
For **decoding strategies** , we tested both **greedy** decoding and **sampling**.
The purpose of establishing baselines is twofold: first, we aimed to evaluate
each model’s core capabilities without applying any additional techniques, en-
suring an accurate assessment of their performance, as prior studies have shown
varying results with the same models. Second, we intend for our techniques
to enhance these baseline outcomes, which requires rigorous comparison across
multiple iterations, to measure the improvements effectively. To ensure the repro-
ducibility and robustness of our results, we repeated the tests for each Sampling
model configuration using five random seeds, reporting the **maximum average
score** from these runs, as in the original task leaderboard teams were allowed to
submit several runs and kept the highest score from them. While we acknowledge
that small adjustments, particularly to the prompts, could significantly impact
the results, we believe this approach allows for a valid comparative analysis.

**3.3 Reproduced Clinical Reasoning Methods**

Due to the breadth of models and techniques across multiple systems, and the
iterative nature of each approach, reproducing every preliminary experiment in
exhaustive detail would be impractical. Instead, we summarize the core contri-
butions of state-of-the-art methods, presenting their respective scores in Tables

(^1) https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.
(^2) https://huggingface.co/BioMistral/BioMistral-7B
(^3) https://huggingface.co/amazon/MistralLite
(^4) https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct
(^5) https://anonymous.4open.science/r/ECIR-2025_Reproducibility_NLI4CT-8D58/
src/prompts/README.md


6 Anonymous Author(s)

3 and 4. Given the primary goal of reproducing, rather than replicating, we ap-
plied small but impactful modifications to each system, resulting in significant
performance differences.
Gema et al. [10] employed frozen models to evaluate various settings: ICL
(1-shot and 2-shot), CoT, and combined approaches (CoT + 1-shot, CoT + 2-
shot), with reasoning chains generated via Chat-GPT. They also incorporated
Parameter-Efficient Fine-Tuning (PEFT) with zero-shot prompting. In repro-
ducing their work, we applied the following adjustments: we replaced their ICL
+ CoT reasoning chains with those extracted by Liu and Thoma[23] from GPT-4,
selected alternate models (e.g., switching from LLaMa2-13B to LLaMa3.1-8B-
instruct and from GPT-4 to Gemini), utilized a different method for response
label extraction, and adjusted the PEFT hyper-parameters slightly.
For Lee et al.[21], PEFT was employed to train on a synthetic set of data
augmented by Chat-GPT 3.5. Additionally, Open-Chat[33] was used to revert
variations of statements to their original forms before processing. Since this
work did not provide open-source code, we focused instead on assessing the
performance of other experimental setups.
In the work by Liu and Thoma [23], PEFT was applied to fine-tune Mixtral-
8x7B on data synthetically augmented by GPT-4, generated through CoT rea-
soning on the NLI4CT train set. Once the model was fine-tuned, they assessed its
zero-shot, CoT, and CoT + self-consistency performance. For our reproduction,
we made several changes: we used a smaller model (Mistral-7B-instruct-v0.2) to
accommodate resource availability, implemented a different response label ex-
traction method, altered prompt syntax to align with Guimarães et al. [12], and
sampled five reasoning chains instead of their original ten.
Finally, Guimarães et al. [12] evaluated PEFT using manually and synthet-
ically augmented data generated with Mistral-7B-instruct-v0.1. In reproducing
their work, we opted for a more concise prompt structure, modified their PEFT
hyper-parameters, utilized an alternative method for label extraction, and ex-
perimented with decoding strategies, including both greedy decoding and adjust-
ments to sampling parameters. We did not incorporate any additional training
data, using the original augmented data from Guimarães et al. [12].

**3.4 Implementation Details**

We implemented our work entirely in Python, relying primarily on thetorch,
transformers, andhuggingfacelibraries. Thehuggingfaceplatform was used
to obtain all the model versions needed for the experiments.
To maintain consistency with prior studies, we adhered to the original prompts
provided by each source, making adjustments when necessary to fit the templat-
ing requirements, or where we managed to improve performance. Instead of repli-
cating the specific pre-processing techniques for CTRs, used by different studies,
we employed the pre-processing functions from Guimarães et al. [12]. Addition-
ally, we did not use the exact same model versions as in the original studies but
opted for updated versions when available. Our goal was not strict replication


```
Consistent LLM Reasoning for NLI over Clinical Trials 7
```
but rather to reproduce the underlying ideas and methodologies, allowing for
modifications in implementation to potentially enhance the results.
For the text generation processes, we handled greedy generation by sim-
ply setting theSampleflag toFalsein the generation. In contrast, for sam-
pling generation, we applied two different configurations based on whether self-
consistency was used. When self-consistency was applied, we settemperature =
1.0,top-p = 0.99, andtop-k = 50. Otherwise, we adjusted these parameters
totemperature = 0.5,top-p = 0.7, andtop-k = 15.
To extract the correct label from the generated response, we first converted
the entire output to lowercase and identified whether the terms associated with
"entailment" or "contradiction" appeared first (e.g., "entailment," "entails," "en-
tailed," etc., for entailment, and "contradiction," "contradicts," "contradicted,"
etc., for contradiction). We assigned the corresponding label unless a negation
word (e.g., "not," "no," "isn’t") was present immediately before these terms, in
which case the label was reversed. Additionally, following the approach of Gema
et al. [10], we reversed the response string for CoT reasoning, allowing us to
search for the last occurrence of “Entailment” or “Contradiction”. This method
reduced label extraction errors, as the model’s final tokens, where the conclusion
is typically expressed, are critical for accurately capturing the intended response
with the context of its own reasoning chain.

## 4 Experimental Methodology

This section describes the methodology employed to reproduce the experiments
and results from the analyzed systems. We detail the key steps taken to ensure
consistency with the original studies, while addressing any challenges that arose
during reproduction.

**4.1 Dataset**

The analyzed CTRs were sourced exclusively from the United States Library
of Medicine^6 , comprising 1,000 English-language trial reports focused on breast
cancer. Each CTR is structured into four sections: **Eligibility Criteria** , **In-
terventions** , **Results** , and **Adverse Events**. The task narrows its focus by
having each clinical statement pertain to one of these sections. An example of a
statement referencing two CTRs is shown in Figure 1.
Ground-truth labels for each Statement-CTR(s) pair are assigned as either
**entailment** , if the CTR(s) fully supports the statement, or **contradiction** , if
the statement is contradicted or lacks supporting information in the CTR(s).
Additionally, each instance is labeled based on whether it refers to one or two
CTRs. **Single** instances involve a direct comparison between the statement and
a single CTR, whereas **comparison** instances may require cross-referencing be-
tween two CTRs. NLI4CT is designed to reflect practical applications, serving
as a foundation for the real-world deployment of models.

(^6) https://clinicaltrials.gov/


8 Anonymous Author(s)

```
Table 1. Full dataset composition for the SemEval-2024 NLI4CT Task
```
```
Split #Pairs
Label (%) Type (%) Section (%)
Entail. Contra. Single Comp. Inter. Elig. Results Adv.Eve.
Train 1700 50.0 50.0 61.0 39.0 23.3 28.6 18.9 29.
Dev 200 50.0 50.0 70.0 30.0 18.0 28.0 28.0 26.
Test 5500 33.5 66.5 46.4 53.6 28.0 25.8 22.5 23.
```
The dataset is divided into **training** , **development** , and **testing** splits,
as detailed in Table 1. The test set includes 5,500 statements, divided into 500
original and 5,000 contrast statements. Contrast statements are variations of the
original ones, generated through processes such as paraphrasing, text appending,
or contradiction. These variations are deliberately crafted to potentially change
the original labels.

**4.2 Metrics**

Through evaluation metrics, the task aimed to emphasize not only the accuracy
of label predictions. but also the resilience of these predictions to variations in
the original statements (an example is presented on Figure 2). To achieve this,
the standard metrics were the following^7 :

**- Macro F1-Score** , serving as a baseline performance metric (not measuring
    robustness), and which is calculated by the arithmetic mean of precision and
    recall, averaged over both possible classes;
**- Faithfulness** , measuring the ability of a system to arrive at the correct
    prediction for the correct reason, which is estimated by calculating the ability
    of a model to change its label prediction when a statement undergoes a
    **semantic-altering** process. For _N_ test set queries _xi_ within the contrast
    set _C_ that corresponds to variations of statement _yi_ , and with _f_ as the
    model’s output for each statement, we define faithfulness as:

```
Faithfulness ( N ) =
```
### 1

### | N |

### ∑ N

```
1
```
```
|f ( yi )−f ( xi )| (1)
```
```
where xi ∈ C ,Label ( xi )̸= Label ( yi )andLabel ( yi ) = f ( yi ).
```
**- Consistency** , measuring the ability of a system to predict the same label
    for equivalent statements, which is estimated by calculating the ability of a
    model to maintain its prediction when a statement undergoes a **semantic-**
    **preserving** process. For _N_ test set queries _xi_ within the contrast set _C_ that
    corresponds to variations of statement _yi_ , and with _f_ as the model’s output
    for each statement, we define consistency as:

```
Consistency ( N ) =
```
### 1

### | N |

### ∑ N

```
1
```
```
1 −|f ( yi )−f ( xi )| (2)
```
(^7) https://sites.google.com/view/nli4ct/semeval-2024/evaluation


```
Consistent LLM Reasoning for NLI over Clinical Trials 9
```
```
where xi ∈ C : Label ( xi ) = Label ( yi )andLabel ( yi ) = f ( yi ).
```
## 5 Experimental Results and Discussion

**5.1 Baseline Results**

Table 2 presents the results for each model discussed in Subsection 3.2, with
different prompt types and text generation methods. Overall, **Mistral-7B** per-
forms well on short prompts, achieving an average score of 69.5 regardless of the
text generation method used. However, its performance drops on long prompts,
with averages of 63.1 for greedy generation and 63.0 for sampling. In con-
trast, **BioMistral-7B** delivers significantly weaker results, particularly with
long prompts. While the short prompt results are moderate, with averages of
57.5 and 55.8, its performance on long prompts is highly inconsistent, marked
by poor F1-scores of 2.4 and 39.3. The model’s high faithfulness scores (99.
and 80.3) can be attributed to it frequently outputting the label "contradiction,"
undermining faithfulness as a reliable metric, as the system fails to adjust to
variations in the input.
Both **MistralLite-7B** and **LLaMa3.1** display more stable performance
across prompt types and generation methods. **MistralLite-7B** maintains av-
erage scores around 61-62 for both prompt lengths, though its F1-scores for
short prompts are notably weak. In contrast, **LLaMa3.1** consistently achieves
the highest performance, with an average score reaching 72.5. Its results remain
steady across both greedy and sampling techniques.
Among the open-source models we evaluated, **Mistral-7B** demonstrates a
solid balance across F1-score, faithfulness, and consistency, while **LLaMa3.**
emerges as the overall strongest baseline. However, the commercial closed-source
**Gemini** models set distinct performance benchmarks, excelling across all met-
rics. In particular, **Gemini-1.5-pro-001** achieves the highest average score,
reaching 82.3 on short prompts with greedy decoding.

**5.2 Experiments without Fine-Tuning**

To reproduce the state-of-the-art results we started by exploring the notable
work of Gema et al. [10] with frozen models, specifically with **Mistral-7B** ,
**LLaMa3.1** , and **Gemini**. Our evaluations spanned the same techniques ex-
plored in the original paper, including zero-shot, in-context examples (1-shot
and 2-shot), chain-of-thought reasoning, and combinations of these methods.
Table 3 presents a comparison between the original results and our reproduced
results, utilizing the official task metrics.
An immediate observation is that we successfully improved consistency across
all scenarios and saw gains in most other metrics across our experiments. The
one exception was our GPT-4 reproduction, where budget constraints required
the use of an alternative commercial model. Nevertheless, Gemini proved to be
a robust baseline, and while Gema et al.’s scores with GPT-4 ranked highest


10 Anonymous Author(s)

```
Table 2. Baseline scores by model, prompt type and decoding technique.
```
```
Model Prompt Decoding Base Metrics Average
F1 Faithfulness Consistency
Mistral-7B-Instruct-v0.2 short Greedy 72.4 70.9 65.2 69.
Mistral-7B-Instruct-v0.2 short Sampling 72.1 71.3 65.0 69.
Mistral-7B-Instruct-v0.2 long Greedy 70.6 56.6 62.0 63.
Mistral-7B-Instruct-v0.2 long Sampling 70.6 56.7 61.7 63.
BioMistral-7B short Greedy 61.3 55.0 56.1 57.
BioMistral-7B short Sampling 64.3 49.4 53.7 55.
BioMistral-7B long Greedy 2.4 99.7* 61.6 54.
BioMistral-7B long Sampling 39.3 80.3* 57.5 59.
MistralLite-7B short Greedy 33.8 89.5 61.6 61.
MistralLite-7B short Sampling 42.5 83.9 58.9 61.
MistralLite-7B long Greedy 58.0 68.4 59.4 61.
MistralLite-7B long Sampling 53.7 68.4 56.9 59.
LLaMa3.1-8B-Instruct short Greedy 59.0 87.5 69.7 72.
LLaMa3.1-8B-Instruct short Sampling 61.7 86.2 69.6 72.
LLaMa3.1-8B-Instruct long Greedy 67.5 80.3 69.2 72.
LLaMa3.1-8B-Instruct long Sampling 67.4 79.6 68.7 71.
Gemini-1.5-flash-001 short Greedy 75.5 86.3 75.2 79.
Gemini-1.5-flash-001 long Greedy 71.6 84.7 71.8 76.
Gemini-1.5-pro-001 short Greedy 76.6 92.6 77.7 82.
Gemini-1.5-pro-001 long Greedy 73.6 92.1 74.1 79.
```
on the leaderboard, our Gemini implementation would have secured a strong
third-place position in the competition
We also observed significant disparities between the original and reproduced
results, particularly in the non-CoT performance of **Mistral-7B**. The orig-
inal study reported low faithfulness scores, which we substantially improved
upon (13.4/70.9, 11.1/88.3, and 13.4/86.3). Additionally, when comparing the
0-shot performance of **Mistral-7B** with the baseline submitted by Guimarães
et al. [12], we observed a modest but still notable improvement.

**5.3 Experiments with Fine-Tuning**

Table 4 presents the results of our Parameter-Efficient Fine-Tuning (PEFT) se-
tups, alongside the original findings. In our reproduction of Gema et al.[10], most
metrics showed improvements with our modifications, though MistralLite-7B ex-
perienced a notable performance drop. In the case of Guimarães et al.[12], our
adjustments significantly boosted faithfulness, though this came at the expense
of F1-score and consistency, keeping average scores close to the original.
For Liu and Thoma’s work [23], while our results were lower overall, it should
be noted that the original model, Mixtral-8x7B, uses a mixture-of-experts con-
figuration with 46.7B parameters—considerably larger than our Mistral-7B-Inst.
Given this difference, our results are within the expected difference range, and
even outperform expectations when adjusted for size disparities.
While we did not re-implement Lee et al. [21], their application of PEFT
with variation reduction techniques shows robust performance, outperforming
our reproduced models and ranking just below GPT-4.


```
Consistent LLM Reasoning for NLI over Clinical Trials 11
```
**Table 3.** Experiments with frozen models, marking _positive_ or _negative_ variations
in terms of F1-Score, Faithfulness, Consistency and their Average measures.

```
Original Model /
Our Model
```
```
Technique Original Results/Our Results
F1 Faithfulness Consistency Average
```
```
Mistral-7B-Inst. [10]/
Mistral-7B-Inst.
```
```
0-shot 65.3/72.4 13.4/70.9 41.5/65.2 40.1/69.
1-shot 66.4/48.8 11.1/88.3 41.3/65.2 39.6/67.
2-shot 66.9/51.1 13.4/86.3 42.5/65.7 40.9/67.
CoT 47.1/48.2 59.3/79.6 50.8/60.5 52.4/62.
CoT+1-shot 58.4/69.4 57.1/84.7 54.9/71.3 56.8/75.
CoT+2-shot 59.4/59.5 60.7/77.5 56.5/63.7 58.9/66.
```
```
LLaMA2-13B [10]/
LLaMa3.1-8B
```
```
0-shot 60.7/67.5 45.0/80.3 49.4/69.2 51.7/72.
1-shot 63.0/69.0 33.5/80.9 48.8/69.1 48.4/73.
2-shot 61.7/70.2 40.2/74.3 50.1/68.1 50.7/70.
CoT 60.3/64.5 50.1/88.2 51.2/68.3 53.9/73.
CoT+1-shot 63.5/69.4 53.1/84.7 53.6/71.3 56.7/75.
CoT+2-shot 59.2/68.5 61.2/87.0 55.5/72.8 58.6/76.
Mistral-7B-Inst. [12]/
Mistral-7B-Inst. 0-shot 67.3/70.6 61.8/56.7 53.8/61.7 60.1/63.
GPT-4 [10]/Gemini 0-shot 77.5 /76.6 94.8 /92.6 77.5/ 77.7 83.3 /82.
```
**5.4 Final Considerations**

Overall, our reproduction efforts were largely successful in validating and even
extending the original studies, albeit requiring additional discussion for the non-
fine-tuned configurations. While we closely matched the original source methods
in terms of generation configurations and overall prompt structures, there were
notable differences in the implementation of key aspects. One such difference
lies in how label outputs were extracted from the generated text. For instance,
Gema et al. [10] identified the label by checking the last token for occurrences
of “entailment” or “contradiction”. Similarly, Liu and Thoma [23] employed a
comparable approach but added terms like “inconclusive” or “undetermined” to
classify uncertain outputs as contradictions. In contrast, Guimarães et al. [12]
detected the label by checking the first occurrence of these tokens, which marked
a significant departure from the method used in the present study.

Moreover, we deviated from the original pre-processing, resulting in discrep-
ancies in how the CTRs were presented across different systems. Additionally, we
adjusted the positioning of special tokens according to the specific templating
guidelines of each system, which we found to have a significant impact, par-
ticularly in examples involving in-context examples. These variations highlight
areas where even minor deviations in implementation can influence outcomes
and provide insights for future reproducibility efforts.


12 Anonymous Author(s)

**Table 4.** Models trained with **PEFT** , marking positive or negative variations in
terms of F1-Score, Faithfulness, Consistency and their Average measures.

```
Original Model /
Our Model
```
```
Technique Original Results/Our Results
F1 Faithfulness Consistency Average
Mistral-7B-Inst. [10]/
Mistral-7B-Inst.
```
- 76.9/78.7 76.6/85.1 71.4/70.8 75.0/78.

```
MistralLite-7B [10]/
MistralLite-7B - 74.8/79.1 87.3/73.8 72.2/70.5 78.1/74.
LLaMA2-13B [10]/
LLaMa3.1-8B - 67.7/ 81.0 77.3/73.6 66.1/73.7 70.4/76.
```
```
Mistral-7B-Inst. [12]/
Mistral-7B-Inst.
```
- 81.2/80.5 72.3/79.1 69.2/70.5 74.2/76.
Manual Aug. 82.9/77.6 76.9/83.4 71.9/69.9 77.2/77.
Synth. Aug. 78.1/77.8 78.0/85.6 71.0/70.8 75.7/78.
Mix Aug. 80.1/80.5 83.1/79.3 72.2/73.8 78.5/77.

```
Mixtral-8x7B [23]/
Mistral-7B-Inst.
```
- 78.7/79.5 81.0/78.9 73.6/70.3 77.8/76.
CoT 78.7/73.9 89.7/84.7 72.2 /69.7 80.2/76.
CoT+SC 80.0/79.0 90.4/84.0 72.9/69.1 81.1/77.
Mistral-7B [21]/- Reduction 76.2/——– 86.1/——– 78.1/——– 80.1/——–
Solar-10.7B [21]/- Reduction 77.9/——– 92.4/——– 80.9/——– 83.7/——–

## 6 Conclusions

This paper focused on reproducing and critically evaluating state-of-the-art
methods for natural language inference on clinical trial reports, using the SemEval-
2024 NLI4CT dataset as the evaluation setting. The key outcome is a robust and
reproducible empirical study. We contributed with a rigorous replication of key
approaches in the field, reinforcing the reliability of existing baselines while offer-
ing improvements in techniques that do not rely on fine-tuning. Furthermore, we
successfully reproduced fine-tuning-based methods, achieving slight performance
gains across various configurations. By fulfilling the central objective of produc-
ing a comprehensive and reproducible analysis, this work lays the groundwork
for future advancements. Our implementations have been made publicly avail-
able^8 , in the hope that they will encourage continued research and refinement
of these methods within the clinical domain.

For future directions, the benchmarking efforts outlined in this study offer
a strong foundation for expanding this research to other datasets within the
clinical domain, such as those from the TREC 2021 Clinical Track, MEDNLI,
and Evidence Inference 2.0, could further extend its impact. Moreover, exploring
the generalization of these techniques to other highly specialized fields, such as
legal NLI [3], represents a promising avenue for further exploration.

(^8) https://anonymous.4open.science/r/ECIR-2025_Reproducibility_NLI4CT-8D58/


```
Consistent LLM Reasoning for NLI over Clinical Trials 13
```
**Disclosure of Interests.** The authors have no competing interests to declare that
are relevant to the content of this article.