Venue: NAACL 2025

# SylloBio-NLI: Evaluating Large Language Models on

Biomedical syllogistic Reasoning

Magdalena Wysocka\({}^{1,2}\), Danilo Carvalho\({}^{1,2}\), Oskar Wysocki\({}^{3}\),

**Marco Valentino\({}^{3}\), Andre Freitas\({}^{1,2,3}\)**

\({}^{1}\)Department of Computer Science, University of Manchester, UK

\({}^{2}\)National Biomarker Centre (NBC), CRUK Manchester Institute, UK

\({}^{3}\) Idiap Research Institute, Martigny, Switzerland

###### Abstract

Syllogistic reasoning is crucial for Natural Language Inference (NLI). This capability is particularly significant in specialized domains such as biomedicine, where it can support automatic evidence interpretation and scientific discovery. This paper presents SylloBio-NLI1, a novel framework that leverages external ontologies to systematically instantiate diverse syllogistic arguments for biomedical NLI. We employ SylloBio-NLI to evaluate Large Language Models (LLMs) on identifying valid conclusions and extracting supporting evidence across 28 syllogistic schemes instantiated with human genome pathways. Extensive experiments reveal that biomedical syllogistic reasoning is particularly challenging for zero-shot LLMs, which achieve an average accuracy between 70% on generalized modus ponens and 23% on disjunctive syllogism. At the same time, we found that few-shot prompting can boost the performance of different LLMs, including Gemma (+14%) and LLama-3 (+43%). However, a deeper analysis shows that both techniques exhibit high sensitivity to superficial lexical variations, highlighting a dependency between reliability, models' architecture, and pre-training regime. Overall, our results indicate that, while in-context examples have the potential to elicit syllogistic reasoning in LLMs, existing models are still far from achieving the robustness and consistency required for safe biomedical NLI applications.

Footnote 1: code and dataset available at: [https://github.com/neuro-symbolic-ai/SyllioBio-NLI](https://github.com/neuro-symbolic-ai/SyllioBio-NLI)

_Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 7235–7258

April 29 - May 4, 2025 <2025 Association for Computational Linguistics

## 1 Introduction

Syllogistic reasoning - i.e., the process of deriving valid conclusions from premises through the systematic application of abstract reasoning schemes - is a fundamental type of inference for developing and evaluating Natural Language Inference (NLI) models that can reason over textual evidence at scale MacCartney and Manning (2009); Wu et al. (2023). within specialised domains such as biomedicine, the ability to reason over natural language can have significant practical implications for supporting complex discourse interpretation, scientific discovery, and the development of downstream biomedical and clinical applications Jullien et al. (2023, 2024), allowing for a set of formally defined patterns for controlled inference.

Moreover, determining the reliability and robustness of NLI models with regard to syllogistic reasoning is a crucial type of assessment, particularly in critical domains requiring safety guarantees Eisape et al. (2024); Jullien et al. (2024). This type of evaluation is further motivated by two key factors. First, syllogistic reasoning is prevalent in discourse at large and, therefore, it is expected that approaches based on Large Language Models (LLMs) are exposed to a variety of reasoning schemes during pre-training Wu et al. (2023). Second, a model that learns to perform syllogistic

Figure 1: End-to-end diagram of the proposed methodological framework, illustrating the generation of syllogistic arguments from domain-specific ontologies, parameterized input to LLMs, and evaluation tasks including textual inference and premise selection.



reasoning should intrinsically possess the ability to generalize to different domains, regardless of the specific world knowledge acquired during pre-training Kim et al. (2024). This is because the ability to perform syllogistic reasoning should be content-independent, that is, the ability to derive logically valid conclusions is only a function of the formal logical schemes and should be independent of its concrete instantiation. However, despite the importance of syllogistic reasoning for NLI and the abundance of benchmarks involving commonsense knowledge Yu et al. (2023), resources for assessing how systematic reasoning capabilities transfer to specialised domains are still scarce and require substantial expert-level annotation effort to guarantee quality and correctness Zhao et al. (2024); Eisape et al. (2024); Porada et al. (2022); Wysocka et al. (2024).

This paper focuses on advancing the availability of resources for biomedical syllogistic reasoning along with our understanding of the capabilities of state-of-the-art NLI models. In particular, we propose SylloBio-NLI, a novel framework for automatically generating NLI resources for evaluating syllogistic reasoning within biomedical domains. Specifically, we demonstrate how external domain-specific resources such as ontologies and thesauri can be leveraged to instantiate a wide range of syllogistic schemes for biomedical NLI tasks, including textual entailment and evidence extraction.

The methodological framework behind SylloBio-NLI is designed to address the annotation scarcity problem for the granular evaluation of complex reasoning in specialized domains. The framework minimises the human annotation effort while guaranteeing the correctness of the generated data by leveraging explicit domain knowledge in the ontologies and the systematicity of known syllogistic schemes (Fig. 1).

By instantiating SylloBio-NLI on human biological pathways using Reactome Croft et al. (2014), we evaluate the domain-specific reasoning capabilities of 8 open-source LLMs on 28 syllogistic schemes, comparing the performance in a zero-shot (ZS) and few-shot (FS) settings. An extensive empirical evaluation led to the following findings and conclusions:

1. We determine that LLMs exhibit surprisingly low ZS performance on biomedical syllogistic arguments with an average accuracy between 70% on _generalised modus ponens_ and 23% on _disjunctive syllogism_ (where random performance is equal to 50%). These low performances are generally shared across models' family (i.e., Llama-3, Mistral, Gemma, BioMistral) and pre-trained regimes (language modelling and instruction-tuning).

2. We found that the FS setting can improve the performance of different LLMs. In particular, we observe a significant boost for both Gemma and Llama-3, with an average increase in F1-score of 14% and 43% respectively. At the same time, the experiments reveal that such improvement is inconsistent across models' families and pre-training regimes.

Figure 2: Domain-specific pipeline for creating natural language instances of argument schemes with multiple templating. Steps include selecting a syllogistic schema (A), applying a domain-specific template (B), instantiating with predicates and entities (C), permuting premises (D), and prompting LLMs for evaluation (E).

 

3. We perform a robustness analysis adopting logically equivalent variations of the same syllogistic schemes by rephrasing the arguments via negations, complex predicates, and De Morgan's laws. Such analysis reveals that both ZS and FS techniques are highly sensitive to surface-form and lexical variations, demonstrating a shared inability to systematically abstract the underlying reasoning rules required to derive valid conclusions. These results indicate that, while FS has the potential to elicit syllogistic reasoning in LLMs, existing models are still far from achieving the robustness and consistency required for safe biomedical NLI.

Overall, the above findings suggest that, upon granular inference scrutiny, the reasoning mechanisms induced in LLMs still confound formal and material inference patterns. Moreover, while there are FS intervention mechanisms which can improve models' performance, delivering controlled specialised syllogistic reasoning remains a challenge for LLMs at large.

To the best of our knowledge, this is the first work focusing on designing a methodology for evaluating syllogistic reasoning within specialised domains, thoroughly assessing the performance of LLMs, and releasing a domain-specific resource for supporting future work in the field. The code for the dataset generation and the evaluation pipeline is fully available online1.

Footnote 1: [https://github.com/facebook/favid-2018/](https://github.com/facebook/favid-2018/)

## 2 SylloBio-NLI

We introduce a general framework for developing resources to systematically evaluate the syllogistic reasoning capabilities of NLI models within biomedical domains. In particular, SylloBio-NLI leverages the systematicity of known syllogistic schemes Betz et al. (2021) along with externalhesauri/ontologies to scale up the generation of deductively valid arguments and to characterise inference performance across a granular set of syllogistic schemes.

To this end, we focus on seven general syllogistic schemes, including _generalized modus ponens, generalized contraposition_, _hypothetical syllogism 1_, _hypothetical syllogism 3_, _generalized modus tollens_, _disjunctive syllogism_ and _generalized dilemma_. These base schemes are then adopted to generate 28 different variations of syllogistic argument templates applying negation, complex predicates, and De Morgan's laws, which can then be instantiated with concrete domain knowledge (see Fig. 6 in Appendix B). The overall methodology is outline by the stages in Fig. 2.

First, for each syllogistic scheme, a corresponding formal argument scheme (consisting of abstract premises and conclusion expressed in first-order logic) is created (Fig. 2A). Next, each symbolic formula in the formal scheme is individually replaced by a natural language domain-specific sentence schema (Fig. 2B). For example, the formulae for the base schema of generalized modus ponens (i.e., Premise 1: \(\forall xFx\Rightarrow Gx\), Premise 2: \(Fa\), Conclusion: \(Ga\)) can be translated into the natural language template: _"Premise 1: Every member of F is a member of G"_. Premise 2: _"a is a member of F"_. Conclusion: _"a is a member of G"_.

Subsequently, the entity and property placeholders in the natural language template are replaced argument-wise with domain-specific entities and predicates extracted from an external ontology (Fig. 2C). In this process, the syllogistic schemes provide the logical validity component of the formal inference while the ontology subdomain and its associated mapping to the schemes provide the soundness (content-based) for the premises and conclusions. Hence, we obtain instances of syllogistic arguments in natural language that are stored in a knowledge base of premises and conclusions.

Finally, the natural language syllogistic arguments can be structured to create prompts for evaluating LLMs. This involves introducing the syllogistic argument, clearly framing the premises, and specifying the instructions for the inference, as illustrated in Appendix F.

### Biomedical NLI Tasks

SylloBio-NLI takes advantage of the corpus's design, where every premise and conclusion in each argument is explicitly stated and fully transparent (Fig. 2). Once the full set of natural language syllogistic arguments is generated, the premises and conclusions in each instance can be adopted to define two different NLI tasks:

**Task 1.** A _textual entailment_ task where the model determines whether the conclusion logically follows from the premises, with the output being 'True' or 'False'. The task is designed using natural language syllogisms generated from formal schemes, with valid and invalid argument instances. The model evaluates each argument's logical validity, requiring it to reason deductively based on the given premises. The focus is on testing the ability 

to discern valid reasoning structures in a biomedical context, independent of factual correctness.

Formally, given a set of premises \({\cal P}=\left\{P_{1},P_{2},\ldots,P_{n}\right\}\) and a conclusion \(C\), the task is to determine whether the logical entailment holds:

\[{\cal P}\models C\]

such that (i) The premises and conclusion are instantiated from formal syllogistic schemes and (ii) arguments may be _valid_ or _invalid_. The assessment is purely on logical validity, independent of the factual correctness of premises and conclusion.

**Promises:** A set \({\cal P}=\left\{P_{1},P_{2},\ldots,P_{n}\right\}\) of premises, where each \(P_{i}\) is a natural language sentence derived from syllogistic schemes.

Conclusion:A natural language sentence \(C\), representing the conclusion.

Logical Validity Indicator:A binary output Output \(\in\left\{\text{True},\text{False}\right\}\) such that:

\[\text{Output}=\begin{cases}\text{True}&\text{if ${\cal P}\models C$},\\ \text{False}&\text{if ${\cal P}\not\models C$}.\end{cases}\]

Here, \({\cal P}\models C\) denotes that \(C\) is a logical consequence of \({\cal P}\).

Task 2.A _premise selection_ task where the model has to identify which premises are necessary and sufficient to justify the conclusion from Task 1, noting that some premises may be irrelevant. The task involves presenting the model with a set of premises, including both relevant and distractor premises. The model must correctly select the subset of premises that logically support the conclusion, excluding those that do not contribute to the entailment. This tests the ability to filter essential information and assess the understanding of the logical relationships between premises and conclusions. Task 2 can be formalized as follows:

Objective:Identify the minimal subset of premises \({\cal P^{\prime}}\subseteq{\cal P}\) that are necessary and sufficient to logically derive the conclusion \(C\).

* A set of premises \({\cal P}=\left\{P_{1},P_{2},\ldots,P_{n}\right\}\).
* A conclusion \(C\).
* **Task Definition:** Find the minimal subset \({\cal P^{\prime}}\subseteq{\cal P}\) satisfying: 1. **Logical Sufficiency:** \[{\cal P^{\prime}}\models C.\]
2. **Minimality:** \[\nexists\,{\cal P^{\prime\prime}}\subsetneq{\cal P^{\prime}},\;\text{such that ${\cal P^{\prime\prime}}\models C$}.\]
3. **Necessity:** Every premise in \({\cal P^{\prime}}\) is necessary for the entailment.

**Premises:** A set \({\cal P}=\left\{P_{1},P_{2},\ldots,P_{n}\right\}\) containing both relevant and irrelevant (distractor) premises.

Conclusion:A natural language sentence \(C\).

* **Selected Premises:** A subset \({\cal P^{\prime}}\subseteq{\cal P}\) such that: \[{\cal P^{\prime}}\models C,\] and \({\cal P^{\prime}}\) is minimal and necessary.

### Human Genome Pathways for Syllogistic Reasoning Evaluation

To instantiate the SylloBio-NLI methodology, we developed a specialized dataset using Reactome2, a comprehensive knowledge base containing detailed information on human biological pathways, including gene functions and interactions.

Footnote 2: [https://reactome.org](https://reactome.org)

Reactome's hierarchical structure, with its well-defined hierarchical gene-pathway membership relations, enables a systematic instantiation of the syllogistic arguments. This allows for the efficient generation of domain-specific NLI tasks that assess LLMs' ability to reason about biological pathway membership, controlling for different levels within the hierarchy. This use case is widely relevant in the context of interpreting pathway-level interactions, disease and treatment response mechanisms at a genomics level (Fang et al., 2019).

By focusing on the Disease super-pathway, the largest and most intricate group within Reactome, we generated a diverse set of syllogistic schemes (see Fig. 2) that mirror the complex molecular interactions and regulatory mechanisms underlying disease processes. This approach ensures our evaluation reflects the depth and specificity needed to rigorously test the syllogistic reasoning properties of NLI models in this highly specialised and clinically significant 

and systematic assessment of these properties is essential if LLMs are to be applied to this domain.

The resulting dataset includes 12,098 entity-gene names, which are used as substitutes for _entity_ placeholders. In contrast, pathway names, derived from different levels of biological hierarchies, are treated as _predicate_ placeholders because they describe relationships or actions involving the entities, rather than being entities themselves. The corpus includes 3767 complex predicates in total. The premises of the natural language argument are randomly re-ordered to mitigate potential biases during the evaluation (Fig. 2D).

### Dictionary of Gene and Pathway Membership

Reactome3 (version 88--March 2024) has entries for 11 226 protein-coding genes involved in 15 212 human reactions annotated from 38 549 literature references. These reactions are grouped into 2 698 pathways collected under 29 superpathways (e.g. Immune System) that describe normal cellular functions. Each superpathway is represented as a roughly circular 'burst,' with the central node corresponding to the top-level of the Reactome event hierarchy and concentric rings representing increasingly specific levels of the event hierarchy (sub-pathways) (e.g. Disease \(\rightarrow\) Diseases of signal transduction by growth factor receptors and second messengers \(\rightarrow\) Signalling by EGFR in Cancer \(\rightarrow\) Signalling by Ligand-Responsive EGFR Variants in Cancer \(\rightarrow\) Constitutive Signalling by Ligand-Responsive EGFR Cancer Variants). The relationships between these pathways are captured through parent-child arcs, reflecting the ontological "is-a" relationships.

Footnote 3: [https://reactome.org](https://reactome.org)

The 29 Reactome superpathways group are each organized as a roughly circular 'burst'. However, we built the corpus based on one, largest group of pathways called Disease. The central node of the Disease burst corresponds to the uppermost level of the Reactome event hierarchy (Table 1). Concentric rings of nodes around the central node represent successive more specific levels of the event hierarchy (e.g. Disease \(\rightarrow\) Diseases of signal transduction by growth factor receptors and second messengers \(\rightarrow\) Signalling by EGFR in Cancer \(\rightarrow\) Signalling by Ligand-Responsive EGFR Variants in Cancer \(\rightarrow\) Constitutive Signalling by Ligand-Responsive EGFR Cancer Variants). The arcs connecting nodes between successive rings within a burst represent parent-child (is-a) relationships in the event hierarchy. When a specific pathway is shared by more than one burst, arcs connect its nodes between bursts. A node's size is proportional to the number of physical entities (proteins, complexes, chemicals) it contains.

### Reasoning Challenges

**Domain-specific Reasoning.** Biomedical datasets differ from general datasets in that they involve complex semantic structures, such as gene-pathway relationships and hierarchies, where sentences encode intricate biological interactions and dependencies. Reasoning about statements such as _"COLIA1 is involved in the TGF-beta signalling pathway"_ requires not only an understanding of specific gene functions but also the ability to navigate over hierarchical biological pathways and their associated relations. This demands a higher level of domain-specific reasoning and the capability to controllably interpret relationships within a highly specialised context.

Material Inference Component.Each sentence in the formal scheme (premises and conclusion) is linked to the domain knowledge, i.e. the membership of genes to a given level of pathway, which refers to a multi-level, hierarchy. For example, if gene X is a member of pathway Y, and that pathway is a child of a top-level pathway Z, the model must be able to infer that the gene also belongs to a top-level pathway.

Formal Inference Component.Syllogisms are examples of content-independent formal inference in which, provided the truth value of the premises, the conclusion can be derived through the application of specific logical inference rules. This form of reasoning can pose challenges to models that are incapable of abstracting inference patterns from text and that are affected by content biases in their representation [17, 1]. Moreover, the syllogistic schemes in this paper

\begin{table}
\begin{tabular}{l c c} \hline \multirow{2}{*}{**Level of pathway**} & **Nr of unique** & **Nr of unique** \\  & **pathway names** & **genes names** \\ \hline top-level & 1 & \\ sub-pathway & 13 & 2131 \\ sub-sub-pathway & 59 & 2131 \\ sub-sub-sub-pathway & 345 & 2114 \\ sub-sub-sub-sub-pathway & 1110 & 1775 \\ sub-sub-sub-sub-sub-pathway & 994 & 1002 \\ \hline \end{tabular}
\end{table}
Table 1: Summary of a dictionary with the true taxonomic relationships between pathways and genes.

 

are designed to assess the interpretation of fine-grained logical operators including quantifiers, implications and negation, which have been proved to be particularly challenging for NLI models (Pitler et al., 2023).

## 3 Empirical Evaluation

### Model Architectures

We used the proposed methodology and resources to assess the syllogistic NLI inference properties of eight open-source LLMs.

Specifically, We test a range of architectures, including mistralai/Mistral-7B-v0.1, mistralai/Mistral-7B-Instruct-v0.2 (Jiang et al., 2023), mistralai/Mixtrl-8x7B-Instruct-v0.1 (Mistral, 2023), google/gamma-7b, google/gemma-7b-it (Gemma and Google, 2024), meta-llama/Meta-Llama-3-8B, meta-llama/Meta-Llama-3-8B-Instruct (AI@Meta, 2024), BioMistral/BioMistral-7B (Labrak et al., 2024). Details of the models, access, parameters, and the prompts used are available in the Appendix C, D, F, G.

### Evaluation Metrics

Accuracy, F1-score, Recall, and Precision are used to evaluate performance in Task 1, a binary classification task where the conclusion \(C\) is labelled as either True or False. These metrics compare the predicted labels \(\hat{y}_{i}\) against the annotated gold labels \(y_{i}\) (True/False).

We used Reasoning Accuracy \(RA\) to evaluate the models' performance in Task 2 to correctly predict the entailment label \(\hat{y}_{i}\) and select the appropriate subset of premises \(\hat{P}_{i}\) that justify this prediction. The metric is calculated as the percentage of responses where both the predicted label \(\hat{y}_{i}\) matches the ground truth \(y_{i}\) and the selected premises \(\hat{P}_{i}\) match the ground truth premises \(P_{i}\). In addition, we use a measure of faithfulness introduced by Jullien et al. (2024), which assesses the ability of a model to correctly change its prediction when altering the truth value of the conclusion (see Appendix E for additional details).

### Main Results and Discussion

The main results on Task 1 and 2 are reported in Table 2. In particular, it is possible to derive the following main conclusions:

**ZS LLMs struggle with controlled, domain-specific syllogistic inference.** Overall, the results demonstrate that LLMs exhibit significant challenges when performing biomedical syllogistic reasoning in a ZS setting (Table 2). We found, in fact, that the majority of the models struggle to achieve performances that are above the 50% random accuracy on Task 1, with Gemma-7B-it being the only exception, achieving an accuracy of 64%. Similarly, in Task 2, Gemma-7B-it was again the best-performing model, achieving _reasoning accuracy_ of 55%. In general, when comparing the performance on the two tasks, we observed that the models achieved lower performance in the premise selection task compared to textual entailment, indicating a shared inability of ZS models to identify the sufficient and necessary set of premises required for the entailment to hold. In addition to low performance, we found a discrepancy between faithfulness and accuracy on both tasks, which

\begin{table}
\begin{tabular}{l c c c c c c c|c c c c c c c c} \hline \multirow{2}{*}{**Model**} & \multicolumn{3}{c|}{**ZS**} & \multicolumn{3}{c}{**FS**} \\  & Non-empty & Increase & Following & **Acc.** & **Recall** & **Precision** & **F1** & **Full.** & non-empty & Intertext & Following & **Acc.** & **Recal** & **Prediction** & **F1** & **Faith.** \\  & compute & text & importance & & & & & & & & & & & & \\ \hline BioMistral-7b & 0.00 & – & – & – & – & – & – & 0.00 & – & – & – & – & – & – \\ \hline Meta-Lima-3-8B & 0.03 & 0.03 & 0.01 & 0.00 & 0.01 & 0.01 & 0.01 & 0.00 & 0.69 & 0.01 & 0.68 & 0.35 & 0.51 & 0.39 & 0.44 & 0.19 \\ Meta-Lima-3-8B-Internet & **1.00** & 0.86 & 0.14 & 0.11 & 0.18 & 0.16 & 0.17 & 0.04 & **1.00** & 0.99 & 0.01 & 0.00 & 0.00 & 0.00 & 0.00 \\ \hline Mixed-7B-instance+0.2 & **1.00** & 0.1 & 0.9 & 0.47 & 0.63 & 0.48 & 0.45 & **1.00** & 0.24 & 0.76 & 0.45 & 0.44 & 0.45 & 0.44 & **0.46** \\ Miniari-7B-v0.1 & 0.58 & 0.88 & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 & 0.69 & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 \\ Maxd-8X7B-features+0.1 & **1.00** & 0.37 & 0.63 & 0.52 & 0.45 & 0.53 & 0.49 & 0.35 & **1.00** & 0.09 & 0.91 & **0.61** & 0.44 & **0.67** & 0.53 & **0.46** \\ \hline Gemma-7b & 0.99 & 0.2 & 0.79 & 0.42 & **0.76** & 0.43 & 0.57 & 0.09 & **1.00** & 0.00 & **1.00** & 0.00 & **1.00** & 0.35 & **0.71** & 0.19 \\ Gemma-7b-it & **1.00** & 0.00 & **1.00** & **0.64** & **0.71** & **0.62** & **0.66** & **0.10** & 0.03 & 0.97 & 0.63 & 0.46 & 0.69 & 0.55 & 0.42 \\ \hline \end{tabular} 
\begin{tabular}{l l c c c c c c c c} \hline \multirow{2}{*}{**Model**} & \multicolumn{3}{c}{**ZS**} & \multicolumn{3}{c}{**FS**} \\  & Non-empty & Irrelevant & Following & **R** & **Avish** & Non-empty & Irrelevant & Following & **R** & **Avish** \\  & output & text & instruction & **R** & **Avish** & Non-empty & Irrelevant & Following & **R** & **Avish** \\ \hline BioMistral-7b & 0.00 & 0.00 & – & - & 0.00 & – & - & 0.00 & – & - & - \\ \hline Meta-Lima-3-8B & 0.02 & 0.01 & 0.02 & 0.00 & 0.00 & 0.87 & 0.01 & 0.87 & 0.35 & 0.31 \\ Meta-Lima-3-8B-Internet & **1.00** & 0.36 & 0.64 & 0.30 & 0.33 & **1.00** & 0.99 & 0.01 & 0.01 & 0.00 \\ \hline Montreal-7B-instance+0.2 & **1.00** & **0.02** & **0.98** & 0.26 & 0.64 & **1.00** & 0.05 & 0.95 & 0.34 & 0.65 \\ Multi-7B-1.0 & **1.00** & **1.00** & 0.00 & 0.00 & 0.00 & 0.04 & 0.84 & 0.00 & 0.00 & 0.00 \\ Microtal-8x7B-Intrnet+0.1 & **1.00** & 0.05 & 0.95 & 0.34 & 0.65 & **1.00** & 0.05 & 0.95 & 0.24 & 0.46 \\ \hline Gouma-7b & 0.69 & 0.31 & 0.38 & 0.23 & 0.14 & **1.00** & 0.00 & **1.00** & **0.58** & **0.49** \\ Gumm-7b-it & **1.00** & 0.03 & 0.97 & **0.55** & **0.65** & **1.00** & 0.00 & **1.00** & 0.55 & 0.31 \\ \hline \end{tabular}
\end{table}
Table 2: Main results for Task 1 (i.e., textual entailment, top) and Task 2 (i.e., premise selection, bottom).

 

highlights the inability to capture the underlying reasoning phenomena along with the presence of biases in the inference process. For example, we observed that for Gemma-7B, the ZS accuracy and F1-score are \(\approx\) 0.5-0.6, while faithfulness plummets below 0.1 (Fig. 3). This is due to the tendency of the model to frequently output "True" (see Figs. 8-11 in Appendix K and Appendix H) regardless of the truth value of the conclusion. In contrast, we observed that such a phenomenon is less prominent in Gemma-7B-it where faithfulness is comparable to accuracy on both Task 1 (i.e., 0.66) and Task 2 (0.65). Overall, the results reveal that instruction-tuned models can obtain higher performances in the ZS setting, while models that are pre-trained on language modelling struggle to follow the provided natural language instructions. Moreover, we observed that this trend is independent of the specific knowledge the LLM is exposed to during pretraining. For instance, BioMistral-7B, the only biomedical domain-specific model, entirely failed to follow the instructions and generate outputs that are relevant to the target tasks.

FS can improve performance, but its impact is inconsistent across models' families and pretraining regimes.We found that the FS setting can improve the performance of different LLMs. In particular, we observe a significant boost for both Gemma-7b and Meta-Llama-3-8b, with an average increase in F1-score of 14% and 43% respectively. However, we observe such improvement to be inconsistent across models' family and pre-training regime, indicating variability in how different models handle prompt types in syllogistic reasoning tasks within the biomedical domain (Fig. 4). In particular, the results show that FS improves for all schemes only for Gemma-7b and Llama-3. For Mixtral-8x7B Instruct, Mistral-7B-Instruct, and Gemma-7b-it the overall _accuracy_ increases only for a subset of selected schemes, while for others it drops significantly. This inconsistency is demonstrated by the fact that Mixtral-8x7B Instruct achieves worse accuracy in FS compared to ZS on both tasks. Overall, the results on the FS setting highlight distinct strengths and weaknesses that are highly dependent on the prompt

Figure 4: Accuracy across two prompt types: ZS and FS, for Task 1 (top) and Task 2 (bottom). The lines connect the average accuracy for each of the seven syllogistic argument schemes, with green lines indicating an increase and red lines indicating a decrease. Gray boxplots display the median, Q1, Q3, and minimum and maximum values.

Figure 3: Comparative Analysis of accuracy, F1, and Faithfulness across two prompt types: ZS (left) and FS (right) for Task 1 for the four best models.

 

strategy, the pre-training setup, and the model's family. At the same time, improvements observed in FS for some models might indicate a potential for enhanced performance when contextual information is available, although this gain is not universal across the models.

Significant variability in model accuracy is observed across different syllogistic schemes.In general, we found that LLMs exhibit variable performances depending on the specific syllogistic schema used for evaluation. The results indicate that LLMs perform better on _generalized modus ponens_ (i.e., three out of four models), with _accuracy_ levels ranging from 0.71 to 0.98 in the FS setting in Task 1 (Fig. 5, Table 5 in Appendix L) and with _reasoning accuracy_ scores ranging from 0.56 to 0.74 in Task 2 (Table 7 in Appendix L). On the other side, In the ZS setting, the Gemma-7B-it model achieved higher performance on _generalized modus tollens_ (accuracy = 0.78 in Task 1 and RA = 0.74 in Task 2), while in the FS setting, the highest performances were registered on _generalized dilemma_ in Task 1 (accuracy = 0.85) and _generalized modus ponens_ in Task 2 (RA = 0.74).

### Robustness to Lexical Variations

We performed a robustness analysis adopting surface-form variations applied to the syllogistic schemes. This is done by rephrasing the natural language syllogistic arguments via negation, complex predicates, and De Morgan's laws, keeping the underlying logical relation between premises and conclusion unaltered (see Fig. 6 in Appendix B). The results of the analysis are reported in Figure 5.

This intervention reveals that LLMs are highly sensitive to surface forms and lexical variations, showing a significant variability across different schemes (Fig. 5). At the same time, we found that in the FS setting, model responses demonstrated overall greater consistency, with lower fluctuations across variants.

In Task 1, _disjunctive syllogism_ was generally identified as the most challenging scheme in the ZS setting for all four models, with none exceeding an accuracy of 0.35, significantly worse than random guessing. While FS slightly improved accuracy, disjunctive syllogism remained the most challenging scheme. Overall, only Gemma-7B in the FS setting effectively handled all syllogistic schemes and their variants in Task 1, demonstrating a higher level of consistency. The other models exhibited

Figure 5: Robustness to lexical variations. The graphs show the accuracy by model, syllogistic schema and lexical variations for ZS and FS respectively for Task 1 (top) and Task 2 (bottom).

 

significant variability depending on the specific surface form, which highlights a shared inability to systematically abstract the underlying reasoning rules required to derive valid conclusions.

### Impact of Distractors and Factuality

Finally, we perform an analysis introducing an increasing number of distractors (i.e., irrelevant premises) in the syllogistic arguments and replacing existing genes in Reactome with synthetic gene names to generate arguments that are independent of medical content and assess the impact of factuality on reasoning (see Appendix I, J).

We found that LLMs show varied sensitivity to distractors, with models like Gemma-7b exhibiting a significant decline in reasoning accuracy as the number of distractors increases, while Mistral-7B Instruct shows improvements in some cases. The impact is scheme-dependent and reveals that more complex syllogisms are particularly affected by increasing distracting information (Figs. 12-15 in Appendix K).

Furthermore, contrary to previous work showing that LLMs exhibit higher performance on syllogisms that are in line with commonsense knowledge Eisape et al. (2024); Kim et al. (2024), we found stable performances when intervening on the factual correctness of the biomedical arguments. These results indicate that (1) logical structure and contextual information have a greater impact on output generation than biomedical knowledge in both ZS and FS settings and (2) existing models might have limited exposition to human genome pathways information during pre-training, being unaffected by the substitution of real gene names with synthetic ones (Fig. 7 in Appendix J).

## 4 Related Work

Syllogistic reasoning, which involves deriving valid conclusions from given premises based on formal logical structures, has long been a central focus in the study of Natural Language Inference (NLI) Bertolazzi et al. (2024); Kim et al. (2024).

Early transformer models like BERT Devlin et al. (2019) and RoBERTa Liu et al. (2019) were trained on general NLI datasets, such as SNLI Bowman et al. (2015) and MNLI Williams et al. (2018), to address reasoning tasks. However, these models were limited in their ability to generalize abstract reasoning patterns, particularly in more specialized domains. Recent work has explored whether larger, decoder-based language models are capable of capturing syllogistic reasoning without task-specific fine-tuning, as reasoning itself is content-independent Bertolazzi et al. (2024).

Recent research, such as Liu et al. liu et al. (2023), has introduced benchmarks like GLoRE to test logical reasoning in LLMs, utilizing techniques such as Chain-of-Thought (CoT) prompting Wei et al. (2023) and in-context learning (ICL) Huang and Chang (2023). These methods have been shown to improve reasoning performance, but issues remain, as models often rely on superficial patterns instead of deep logical comprehension. Eisape et al. Eisape et al. (2024) further demonstrated that even advanced models exhibit biases when handling syllogisms, particularly when faced with invalid syllogisms, often failing to generate "nothing follows" conclusions.

While the general-domain performance of LLMs in syllogistic reasoning has been explored Eisape et al. (2024); Dasgupta et al. (2024), research on syllogistic reasoning in the biomedical domain remains scarce. This paper addresses this gap by focusing specifically on biomedical applications, where logical reasoning over specialized knowledge is critical. We introduce a large-scale biomedical syllogism dataset, SylloBio-NLI, and provide a comprehensive evaluation of the syllogistic reasoning capabilities of state-of-the-art LLMs within this domain.

## 5 Conclusions

In this work, we proposed a novel methodological framework, SylloBio-NLI, designed to evaluate the syllogistic reasoning capabilities of state-of-the-art LLMs within the biomedical domain. Through comprehensive analysis across 28 syllogistic schemes, we assessed the performance of eight different models under varying conditions, including zero-shot and few-shot settings.

Our results show that both techniques exhibit high sensitivity to superficial lexical variations, highlighting a dependency between reliability, models' architecture, and pre-training regime. Overall, our evaluation indicates that, while few-shot strategies have the potential to elicit syllogistic reasoning in LLMs, existing models are still far from achieving the robustness and consistency required for safe biomedical NLI applications in specialised domains 



## 6 Limitations

A key challenge for scaling our approach to different domains is its dependency on high-quality external ontologies and knowledge bases. This factor limits the scope of our analyses across biomedical domains. More efficient methods for populating the natural language syllogistic arguments could be investigated in future work, involving automated NLP methods, such as those used in RepoDB Brown and Patel (2017), MSI, Hetionet Himmelstein et al. (2017), DrugMechDB Gonzalezes et al. (2023), and INDRA Gyori et al. (2017); Bachman et al. (2023), or synthetic data generation methods coupled with efficient quality checks. However, these approaches still face challenges in balancing precision and generalization, particularly for complex reasoning tasks in biomedicine. Further improvements are necessary to develop scalable resources and more adaptable NLP techniques for real-world applications.

This work was funded by the European Union's Horizon 2020 research and innovation programme under grant agreement No 965397 through the cancer core Europe DART project, by the Swiss National Science Foundation (SNSF) project "NeutMath: Neural Discourse Inference over Mathematical Texts" (200021_204617), by the CRUK National Biomarker Centre, and supported by the Manchester Experimental Cancer Medicine Centre and the NIHR Manchester Biomedical Research Centre.

## References

* A. M. Alta (2024)Llama 3 model card. Cited by: SS1.
* J. A. Bachman, B. M. Gyori, and P. K. Sorger (2023)Automated assembly of molecular mechanisms at scale from text mining and curated databases. Molecular Systems Biology19 (5), pp. e11325. External Links: Document, Link Cited by: SS1.
* L. Bertolazzi, A. Gatt, and R. Bernardi (2024)A systematic analysis of large language models as soft reasoners: The case of syllogistic inferences. Preprint arXiv:2406.11341. Cited by: SS1.
* Gregor Betz, C. Voigt, and K. Richardson (2021)Critical thinking for language models. In Proceedings of the 14th International Conference on Computational Semantics (IWCS), pp. 63-75. Cited by: SS1.
* S. R. Bowman, G. Angeli, C. Potts, and C. D. Manning (2015)A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, Lisbon, Portugal, pp. 632-642. External Links: ISBN 978-3-319

Benjamin M Gyori, John A Bachman, Kartik Subramanian, Jeremy L Muhlich, Lucian Galescu, and Peter K Sorger. 2017. From word models to executable models of signaling networks using automated assembly. _Molecular systems biology_, 13(11):954.
* Himmelstein et al. [2017] Daniel Scott Himmelstein, Antoine Lizee, Christine Hessler, Leo Brueggeman, Sabrina L Chen, Dexter Hadley, Ari Green, Pouya Khankhanian, and Sergio E Baranzini. 2017. Systematic integration of biomedical knowledge prioritizes drugs for repurposing. _Elife_, 6:e26726.
* Huang and Chang [2023] Jie Huang and Kevin Chen-Chuan Chang. 2023. Towards reasoning in large language models: A survey. In _Findings of the Association for Computational Linguistics: ACL 2023_, pages 1049-1065, Toronto, Canada. Association for Computational Linguistics.
* Jiang et al. [2023] Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral ?b. _arXiv preprint arXiv:2310.06825_.
* Jullien et al. [2024] Mael Jullien, Marco Valentino, and Andre Freitas. 2024. SemEval-2024 task 2: Safe biomedical natural language inference for clinical trials. In _Proceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024)_, pages 1947-1962, Mexico City, Mexico. Association for Computational Linguistics.
* Jullien et al. [2023a] Mael Jullien, Marco Valentino, Hannah Frost, Paul O'Regan, Donal Landers, and Andre Freitas. 2023a. NLI4CT: Multi-evidence natural language inference for clinical trial reports. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_, pages 16745-16764, Singapore. Association for Computational Linguistics.
* Jullien et al. [2023b] Mael Jullien, Marco Valentino, Hannah Frost, Paul O'regan, Donal Landers, and Andre Freitas. 2023b. SemEval-2023 task 7: Multi-evidence natural language inference for clinical trial data. In _Proceedings of the 17th International Workshop on Semantic Evaluation (SemEval-2023)_, pages 2216-2226, Toronto, Canada. Association for Computational Linguistics.
* Kim et al. [2024] Geonhee Kim, Marco Valentino, and Andre Freitas. 2024. A mechanistic interpretation of syllogistic reasoning in auto-regressive language models. _arXiv preprint arXiv:2408.08590_.
* Labrak et al. [2024] Yanis Labrak, Adrien Bazoge, Emmanuel Morin, Pierre-Antoine Gournaud, Mickael Rouvier, and Richard Dufour. 2024. Biomisrl: A collection of open-source pretrained large language models for medical domains. _Preprint_, arXiv:2402.10373.
* liu et al. [2023] Hanmeng Liu, Zhiyang Teng, Ruoxi Ning, Jian Liu, Qiji Zhou, and Yue Zhang. 2023. Glore: Evaluating logical reasoning of large language models. _Preprint_, arXiv:2310.09107.
* Liu et al. [2019] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. _ArXiv_, abs/1907.11692.
* MacCartney and Manning [2009] Bill MacCartney and Christopher D Manning. 2009. An extended model of natural logic. In _Proceedings of the eighth international conference on computational semantics_, pages 140-156. Association for Computational Linguistics.
* a high quality sparse mixture-of-experts.
* Pitler et al. [2023] Emily Pitler, Chris Callison-Burch, and Ellie Pavlick. 2023. The logic of learning: Syllogistic reasoning in modern ai systems. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics.
* Porada et al. [2022] Ian Porada, Alessandro Sordoni, and Jackie Cheung. 2022. Does pre-training induce systematic inference? how masked language models acquire commonsense knowledge. In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 4550-4557, Seattle, United States. Association for Computational Linguistics.
* Prange et al. [2023] Jakob Prange, Khalil Mirni, and Noah A. Smith. 2023. Challenges and opportunities in nlp for systematic generalization: Beyond compositionality. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics.
* Wei et al. [2023] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023. Chain-of-thought prompting elicits reasoning in large language models. _Preprint_, arXiv:2201.11903.
* Williams et al. [2018] Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_, pages 1112-1122, New Orleans, Louisiana. Association for Computational Linguistics.
* Wu et al. [2023] Yongkang Wu, Meng Han, Yutao Zhu, Lei Li, Xinyu Zhang, Ruofei Lai, Xiaoguang Li, Yuanhang Ren, Zhicheng Dou, and Zhao Cao. 2023. Hence, socrates is mortal: A benchmark for natural language syllableissoning. In _Findings of the Association for Computational Linguistics: ACL 2023_, pages 2347-2367, Toronto, Canada. Association for Computational Linguistics 

Magdalena Wysocka, Oskar Wysocki, Maxime Delmas, Vincent Mutel, and Andre Freitas. 2024. Large language models, scientific knowledge and factuality: A framework to streamline human expert evaluation. _Journal of Biomedical Informatics_, page 104724.
* Yu et al. (2023) Fei Yu, Hongbo Zhang, Prayag Tiwari, and Benyou Wang. 2023. Natural language reasoning, a survey. _Preprint_, arXiv:2303.14725.
* Zhao et al. (2024) Zehui Zhao, Laith Alzubaidi, Jinglan Zhang, Ye Duan, and Yauntong Gu. 2024. A comparison review of transfer learning and self-supervised learning: Definitions, applications, advantages and limitations. _Expert Systems with Applications_, 242:122807.

## Appendix A Formalization of the SylloBio-NLI Resource Generation Process

This appendix formalises the generation process of the syllogistic inference patterns.

We start by defining the mains constructs (formal and linguistic artefacts and functions) of the underlying framework:

1. **Syllogistic Scheme (\(S\))**: A logical inference pattern consisting of premises and a conclusion, \(S=\{P_{1},P_{2},\ldots,P_{n},C\}\), where \(P_{i}\) is premise \(i\) and \(C\) is the conclusion.
2. **Formal Argument Scheme (\(\sigma\))**: Representation of a syllogistic scheme in first-order logic (FOL), \(\sigma(S)=\{\phi_{1},\phi_{2},\ldots,\phi_{n},\psi\}\), where \(\phi_{i}\) corresponds to \(P_{i}\) and \(\psi\) corresponds to \(C\).
3. **Natural Language Template (\(\tau\))**: A natural language schema mapping each formula in \(\sigma(S)\) to a sentence template, \(\tau(\sigma(S))=\{\tau_{1},\tau_{2},\ldots,\tau_{n},\sigma\}\), where \(\tau_{i}\) is the sentence template for \(\phi_{i}\) and \(\sigma\) is the sentence template for \(\psi\).
4. **Ontology (\(O\))**: A domain-specific knowledge base containing entities \(E\) and predicates \(\Pi\), \(O=\{E,\Pi\}\), where \(E=\{e_{1},e_{2},\ldots,e_{k}\}\) and \(\Pi=\{\pi_{1},\pi_{2},\ldots,\pi_{l}\}\).
5. **Instantiation Function (\(I\))**: A function that replaces placeholders in \(\tau\) with entities and predicates from \(O\), \(I:\tau(\sigma(S))\times O\rightarrow\text{NL}\), where NL is the set of natural language sentences.
6. **Expert Mapping Function (\(\mu_{\text{Expert}}\))**: A function provided by a domain expert to map placeholders to appropriate ontology terms, \(\mu_{\text{Expert}}:\text{Placeholders}\to E\cup\Pi\).
7. **Knowledge Base (KB)**: A collection of instantiated syllogistic arguments, KB \(=\{A_{1},A_{2},\ldots,A_{m}\}\), where \(A_{i}=\{P^{\prime}_{1},P^{\prime}_{2},\ldots,P^{\prime}_{n},C^{\prime}\}\) and \(P^{\prime}_{i},C^{\prime}\) are instantiated natural language sentences.

### Process Formalisation

The process formalisation defines a systematic process for generating domain-specific syllogistic arguments by:

1. **Defining** formal representations of syllogistic schemes in first-order logic.
2. **Generating** natural language templates from these formal representations.
3. **Mapping** placeholders to domain-specific entities and predicates using an ontology and expert knowledge.
4. **Instantiating** the templates to produce logically valid and semantically sound arguments.
5. **Constructing** a knowledge base for evaluating NLI models.

This ensures that the generated arguments are both logically valid and contextually relevant to the biomedical domain.

**Input:** A set of syllogistic schemes: \(\mathcal{S}=\{S_{1},S_{2},\ldots,S_{m}\}\), an ontology: \(O=\{E,\Pi\}\), an expert mapping function: \(\mu_{\text{Expert}}\).

**Output:** A knowledge base of instantiated arguments: KB.

**Step 1: Formal Argument Scheme Selection:** For each syllogistic scheme \(S_{i}\in\mathcal{S}\), define its formal argument scheme in first-order logic:

\[\forall S_{i}\in\mathcal{S},\quad\sigma(S_{i})=\{\phi^{i}_{1},\phi^{i}_{2}, \ldots,\phi^{i}_{n},\psi^{i}\}.\]

**Step 2: Natural Language Template Generation:** Transform each formula in \(\sigma(S_{i})\) into a natural language template:

\[\forall\phi^{i}_{j}\in\sigma(S_{i}),\quad\tau^{i}_{j}=\tau(\phi^{i}_{j}),\]

\[\sigma^{i}=\tau(\psi^{i}).\]

**Step 3: Ontology Mapping and Instantiation:** Apply the expert mapping function to select appropriate entities and predicates from the ontology 

\(\forall\)Placeholder \(p\in\{\mathbf{F},\mathbf{G},\mathbf{a}\},\quad\mu_{\text{Expert}}(p)\to E\cup\Pi\).

Instantiate the templates:

\[P^{\prime}_{j}=I(\tau^{i}_{j},\mu_{\text{Expert}}),\] \[C^{\prime}=I(\sigma^{i},\mu_{\text{Expert}}).\]

under the following constraints:

* **Logical Validity**: The instantiated arguments must preserve the logical structure of \(\sigma(S_{i})\).
* **Domain Soundness**: The selected entities and predicates must be semantically coherent within the targeted subdomain.

These constraints can be further formalised as:

**Logical Validity Constraint:** The instantiated argument \(A_{i}\) must be logically valid:

\[\{\phi^{\prime}_{1},\phi^{\prime}_{2},\ldots,\phi^{\prime}_{n}\}\models\psi^{ \prime},\]

where \(\phi^{\prime}_{j}\) corresponds to the logical form of \(P^{\prime}_{j}\).

**Domain Soundness Constraint:** The entities and predicates used must be semantically valid within the domain:

\[\forall e\in E^{\prime},\pi\in\Pi^{\prime},\quad\textrm{DomainValid}(e,\pi)= \textrm{True},\]

where \(E^{\prime}\subseteq E\) and \(\Pi^{\prime}\subseteq\Pi\) are entities and predicates used in \(A_{i}\).

**Verification of Logical Validity:** Ensure that the instantiated premises logically entail the conclusion:

\[\{\phi^{\prime}_{1},\phi^{\prime}_{2},\ldots,\phi^{\prime}_{n}\}\models\psi^{ \prime},\]

using logical inference rules.

**Verification of Domain Soundness:** Confirm that:

* All entities and predicates are correctly used.
* There are no semantic contradictions.

**Step 4: Knowledge Base Construction:** Aggregate all instantiated arguments into the knowledge base:

\[\textrm{KB}=\bigcup_{i=1}^{m}\{A_{i}\},\]

where:

\[A_{i}=\{P^{\prime}_{1},P^{\prime}_{2},\ldots,P^{\prime}_{n},C^{\prime}\}.\]

This is summarised with the following algorithmic outline:

1. Initialize KB \(\leftarrow\emptyset\).
2. For each \(S_{i}\in\mathcal{S}\): 1. Create \(\sigma(S_{i})=\{\phi^{i}_{1},\phi^{i}_{2},\ldots,\phi^{i}_{n},\psi^{i}\}\). 2. Generate \(\tau(S_{i})=\{\tau^{i}_{1},\tau^{i}_{2},\ldots,\tau^{i}_{n},\sigma^{i}\}\), where \(\tau^{i}_{j}=\tau(\phi^{i}_{j})\). 3. Obtain mappings \(\mu_{\text{Expert}}\) for placeholders in \(\tau^{i}_{j}\) and \(\sigma^{i}\). 4. Instantiate: \[P^{\prime}_{j}=I(\tau^{i}_{j},\mu_{\text{Expert}}),\] \[C^{\prime}=I(\sigma^{i},\mu_{\text{Expert}}).\] 5. Form argument \(A_{i}=\{P^{\prime}_{1},P^{\prime}_{2},\ldots,P^{\prime}_{n},C^{\prime}\}\). 6. Add \(A_{i}\) to KB: \[\textrm{KB}\leftarrow\textrm{KB}\cup\{A_{i}\}.\]
3. Return KB.

## Appendix B Domain-specific pipeline for creating NL instances

Figure 6: Syllogistic argument schemes used to create a biologically factual argument corpus with domain-specific examples for _generalized modus ponens base scheme_ and _disjunctive syllogism complex predicates scheme_. For each syllogistic scheme, a formal argument scheme (consisted of premises and conclusion (bold)) was provided.

 

* mistralai/Mistral-7B-v0.1
* mistralai/Mistral-7B-Instruct-v0.2
* mistralai/Mistral-8x7B-Instruct-v0.1
* google/gamma-7b
* google/gamma-7b-it
* meta-llama/Meta-Llama-3-8B
* meta-llama/Meta-Llama-3-8B-Instruct
* BioMistral/BioMistral-7B

The pretrained LLM weights are used through the _transformers5_ python library. All models were loaded with standard configurations and their respective default tokenizers, using the _AutoModelForCausalLM_ and _AutoTokenizer_ classes. Additionally, the models were loaded with the options _device_map="auto"_, _torch_dtype=""auto"_attn_implementation="flash_attention_2"_ and _offload_buffers=True_ to make the best use of GPU resources available.

Footnote 5: [https://huggingface.co/docs/transformers/](https://huggingface.co/docs/transformers/)

For the instruction models, the inputs were passed through each model's chat template (with \(tokenizer.apply\_chat\_template\)), so that they would follow the appropriate prompt format. Responses were cleaned of special symbols for evaluation. Table 3 contains the relevant characteristics of all analyzed models.

## Appendix D Experimental Details

The entire experimental setup was implemented as a _python_ code package, consisting of 5 modules:

* _pathways_: defines the ontological relations and operations for biological pathways, as

Figure 6: Syllogistic argument schemes used to create a biologically factual argument corpus with domain-specific examples for _generalized modus ponens base scheme_ and _disjunctive syllogism complex predicates scheme_. For each syllogistic scheme, a formal argument scheme (consisted of premises and conclusion (bold)) was provided.

 

well as the logic to retrieve and transform data from the domain ontology.
* _logic2nl_: translates logic formulas into NL statements (premises, conclusions), using parameterised scheme templates.
* _llm_: provides access to LLMs, and facilitates logging.
* _experiments_: defines all test logic, parameterisation and metrics for the experiments.
* _main_: orchestrates all the tests and aggregates results.

The LLMs were loaded and run using the HuggingFace transformers library (v4.43.3). The prompts were processed directly by the models with _generate_, without sampling.

Parameters were set as follows:

* _Number of premises_ = \(2\): the number of valid premises per instance (factual argumentative text).
* _Max distractors_ = \(5\): maximum number of distractors to be added per instance.
* _Subset size_ = \(200\): maximum number of positive and negative instances for each scheme.
* _Batch size_ = \(20\): number of instances evaluated simultaneously.
* _ICL_: Whether the in-context learning prompt would be used or not.
* _Model_: the LLM to be evaluated.

For both Task 1 and Task 2, each model was analyzed across all 28 syllogistic schemes, with responses evaluated from a total of 11,200 instances (28 schemes x (200 positive + 200 negative)). The exceptions were: _Generalized Modus Ponens_ -complex predicates (18 instances), _Hypothetical Syllogism I_ - base (202 instances) and _Generalized Contraposition_ - negation (202 instances). This is due to the number of factually possible different instances being limited by the ground-truth data in such cases. For both tasks, this amounted to a total of 211,840 instances, with performance comparisons made between ZS and ICL settings, leading to a total of 423,680 prompts.

When examining the impact of distractors, responses were analyzed across five variants of each argument scheme, reflecting different numbers of irrelevant premises (from 1 to 5 distractors). For each scheme and model, this resulted in 1,000 responses (200 prompts per scheme x 5 variants), with the exception of the aforementioned cases. Across all schemes, this totaled 26,480 responses for one model, and 211,840 responses were analyzed for all models for each number of distractors, totaling 1,059,200 responses. Again, comparisons were made between ZS and ICL settings, doubling the previous number.

For both Task 1 and Task 2, performance was compared using actual gene entity names versus synthetic names across two selected schemes. Each scheme was tested with 400 instances (200 positive, 200 negative), resulting in a comprehensive analysis of 12,800 instances per task (200 instances x 2 schemes x 2 entity types x 8 models), providing a thorough comparison of model performance with both factual and synthetic gene names.

The experiments were run on a computer with an AMD EPYC 7413 24-Core CPU, 128GB of available RAM and 2 x NVIDIA A100-SXM4-80GB GPUs.

## Appendix E Evaluation Metrics

**Accuracy, F1-score, Recall, and Precision** are used to evaluate performance in Task 1, a binary

\begin{table} \end{table}
Table 3: Detailed Characteristics of Selected LLMs.

 

classification task where the conclusion \(C\) is labeled as either True or False. These metrics compare the predicted labels \(\hat{y}_{i}\) against the annotated gold labels \(y_{i}\) (True/False).

**Reasoning Accuracy**\(RA\) evaluates a model's ability in Task 2 to correctly predict the entailment label \(\hat{y}_{i}\) and select the appropriate subset of premises \(\hat{P}_{i}\) that justify this prediction. The metric is calculated as the percentage of responses where both the predicted label \(\hat{y}_{i}\) matches the ground truth \(y_{i}\) and the selected premises \(\hat{P}_{i}\) match the ground truth premises \(P_{i}\).

\[\text{RA}=\frac{1}{N}\sum_{i=1}^{N}\mathbb{I}\left(\hat{y}_{i}=y_{i}\text{ and }\hat{P}_{i}=P_{i}\right)\]

In addition we used two supplementary metrics: **Non-empty output** quantifies the frequency with which the model generates any textual response. This metric is crucial, as an empty output, while not providing an incorrect answer, indicates a failure to engage with the prompt. **Irrelevant text** assesses the extent to which the model's output deviates from the expected format, such as failing to adhere to the directive to respond with a straightforward "True" or "False", or a structured statement like "The correct answer is True." This metric identifies instances where the model's response is extraneous or non-compliant with the specified guidelines.

**Faithfulness**\(F\) evaluates a model's ability to consistently adjust its predictions in response to meaningful changes in input. In this analysis, faithfulness is measured by examining \(N\) pairs of prompts that differ only in the 'C' conclusion: one concludes with "it is True that..." (\(y_{i}\)) and the other with "it is False that..." (\(x_{i}\)). A pair is deemed faithful if the model changes its prediction (\(f(x_{i})\),\(f(y_{i})\)) accordingly--shifting from True to False or from False to True between the two prompts. If the model's response to either prompt in a pair is empty or contains irrelevant text that does not follow the instructions, the entire pair is considered unfaithful (\(g=0\)). This criterion ensures that only responses that are meaningful and adhere to the given instructions are evaluated.

\[F=\frac{1}{N}\sum_{i=1}^{N}\mathbb{I}\left(f(x_{i})\neq f(y_{i})\right)\cdot \mathbf{g}(f(x_{i}),f(y_{i}))\]

## Appendix F Prompting LLMs - Zero-shot prompts

### Task 1

prompt = _Suppose you are a specialist with existing knowledge about a signaling and metabolic molecules and their relations organized into biological pathways and processes. Given premises marked with the letter P and the following number and the conclusion marked with the letter C, determine whether the conclusion logically follows from these premises. If the conclusion logically follows from the premises, you need to return 'True'. If the conclusion does not follow logically from the premises, you need to return 'False'. The output should be a single word <True> or <False>._

#### f.2.1 "P1: " + Premise 1

"_P2: " + Premise 2_

"_C:_" + Conclusion

### Task 2

prompt = _Suppose you are a specialist with existing knowledge about a signaling and metabolic molecules and their relations organized into biological pathways and processes. Given premises marked with the letter P and the following number and the conclusion marked with the letter C, determine whether the conclusion logically follows from these premises. If the conclusion logically follows from the premises, you need to return 'True'. If the conclusion does not follow logically from the premises, you need to return 'False'. Specify the premises you used to determine whether the conclusion logically follows from the premises, and only these premises. The output should be a single word <True> or <False> and the numbers of the selected premises after the decimal point, like <True, P1, P2>._

"_P1: " + Premise 1_

"_P2: " + Premise 2_

"_C:_" + Conclusion

## Appendix G Prompting LLMs - Few-shot prompts

### Task 1

Context: _Suppose you are a specialist with existing knowledge about a signaling and metabolic molecules and their relations organized into biological pathways and processes.

 

Instructions: _Given premises marked with the letter P and the following number and the conclusion marked with the letter C, determine whether the conclusion logically follows from these premises._

Relevance: _If the conclusion logically follows from the premises, you need to return 'True'. If the conclusion does not follow logically from the premises, you need to return 'False'._

Constraint: _The output should be a single word <True> or <False>._

Demonstration:

"P1: ""Every member of Diseases of hemostasis pathway is a member of Disease pathway"

"P2: ""Gene GP1BB is a member of Diseases of hemostasis pathway"

"C:" "Gene GP1BB is a member of Disease pathway"

The correct answer is: True

"P1: ""Every member of Infectious disease pathway"

"P2: ""Gene PKQQ is a member of Infectious disease pathway"

"C:" "Gene PKQQ is a member of Infectious disease pathway"

The correct answer is: True

"P1: ""Every member of SLC transporter disorders pathway is a member of Disorders of transmembrane transporters pathway"

"P2: ""Gene AXZY is a member of SLC transporter disorders pathway"

"C:" "Gene AXZY is not a member of Disorders of transmembrane transporters pathway"

The correct answer is: False

"P1: ""Every member of HIV Life Cycle pathway is a member of HIV Infection pathway"

"P2: ""Gene MLLX is a member of HIV Life Cycle pathway"

"C:" "Gene MLLW is a member of HIV Infection pathway"

The correct answer is: False

"P1: ""Every member of ABC transporter disorders pathway is a member of Disorders of transmembrane transporters pathway"

"P2: ""Gene PMLS5 is a member of ABC transporter disorders pathway"

"C:" "It is true that Gene PSMC5 is a member of Disorders of transmembrane transporters pathway"

### Task 2

Context: _Suppose you are a specialist with existing knowledge about a signaling and metabolic molecules and their relations organized into biological pathways and processes._

Instructions: _Given premises marked with the letter P and the following number and the conclusion marked with the letter C, determine whether the conclusion logically follows from these premises._

Relevance: _If the conclusion logically follows from the premises, you need to return 'True'. If the conclusion does not follow logically from the premises, you need to return 'False'. Specify the premises you used to determine whether the conclusion logically follows from the premises, and only these premises._

Constraint: _The output should be a single word <True> or <False> and the numbers of the selected premises after the decimal point, like <True, P1, P2>._

Demonstration:

"P1: " _"Every member of Diseases of hemostasis pathway is a member of Disease pathway"_

"P2: " _"Every member of NS1 Mediated Effects on Host Pathways pathway is a member of Influenza Infection pathway"_

"P3: " _"Gene AABC is a member of Diseases of hemostasis pathway"_

"C:" _"Gene AABC is a member of Disease pathway"_

The correct answer is: True, P1, P3

"P1: " _"Every member of SARS-CoV Infections pathway is a member of Viral Infection Pathways pathway"_

"P2: " _"Every member of Infectious disease pathway is a member of Disease pathway"_

"P3: " _"Gene PKQQ is a member of Infectious disease pathway"_

"C:" _"Gene PKQQ is a member of Infectious disease pathway"_

The correct answer is: True, P2, P3

"P1: " _"Every member of SLC transporter disorders pathway is a member of Disorders of transmembrane transporters pathway"_

"P2: " _"Gene ASZY is a member of SLC transporter disorders pathway"_ 

"C:" _Gene AXZY is not a member of Disorders of transmembrane transporters pathway"_

The correct answer is: False

_"P1:" "Every member of HIV Life Cycle pathway is a member of HIV Infection pathway"_

_"P2:" "Gene MLLX is a member of HIV Life Cycle pathway"_

_"C:" "Gene MLLW is a member of HIV Infection pathway"_

The correct answer is: False

_"P1:" "Every member of ABC transporter disorders pathway is a member of Disorders of transmembrane transporters pathway"_

_"P2:" "Gene PSMCS is a member of ABC transporter disorders pathway"_

_"C:" "It is true that Gene PSMC5 is a member of Disorders of transmembrane transporters pathway"_

## Appendix H Results: Misaligned Instruction-Response

We observed four types of text outputs: those aligned with the instruction (regardless of correctness), empty outputs where no text was generated, incorrect text outputs such as repeated prompts or random content, and outputs resembling Chain-of-Thought (CoT) reasoning that, while potentially containing correct reasoning, did not align with the given instructions (Figs. 8-11). We noticed that BioMistral-7B generated empty outputs in 100% of the cases regardless of the specific settings, while Meta-Llama-3-8B exhibits this behaviour for ZS settings in both tasks. We attribute this observation to safety mechanisms applied during pre-training Labrak et al. (2024), suggesting that domain-specific knowledge about human genome pathways is absent in both models. Similarly, Mistral-7B-v0.1 responses simply repeat the prompt text in 88% of the cases in the ZS settings, and 69% of the cases in FS (Table 2). Moreover, CoT outputs including phrases like e.g. "A nice logical puzzle! Let's break it down step by step..." were particularly common for Meta-Llama-3-8B Instruct, which often ignored the specific instructions to address the task. This behaviour highlights potential biases introduced during instruction-tuning which make the models unable to generalise to domains that are out-of-distribution of the training set.

## Appendix I Results: Ambiguous Impact of Distractors on Reasoning

LLMs show a slight sensitivity to increasing number of distractors \(n_{D}\) in the prompt, with overall accuracy remaining stable (Figs. 12- 15, Table 4).

While some models struggle with increasing \(n_{D}\), others can leverage few-shot learning to mitigate their impact, though the effect is scheme-dependent. Considering Task 1, in the ZS setting (Fig. 12, Table 6), Gemma-7b shows a significant decline in accuracy as \(n_{D}\) increases, particularly in the _generalized dilemma_ (\(r=-0.643,p=0.001\)), _generalized modus ponens_ (\(r=-0.592,p=0.002\)), and _generalized modus tollens_ (\(r=-0.571,p=0.004\)) schemes, indicating a moderate negative correlation. In contrast, in the ZS setting, Mistral-7B Instruct-v0.2 exibits a moderate improvement in accuracy with higher \(n_{D}\) , in the _generalized modus tollens_ (\(r=0.540,p=0.006\)) scheme, reflecting a weak positive correlation overall (\(r=0.333,p<0.001\)). Considering _reasoning accuracy_, in the ZS setting (Fig. 14, Table 8), the Gemma-7b model exhibited a substantial drop as the \(n_{D}\) increased (\(r=-0.951,p<0.001\)), with an initial low accuracy of 0.3 even with \(n_{D}=0\) . The steepest declines were observed in _hypothetical syllogism 1_ (\(r=-1.0,p<0.000\)) and _generalized dilemma_ (\(r=-1.0\), \(p<0.000\)) schemes. For the Gemma-7b-it, the strongest negative correlation between the model and \(n_{D}\) was for the _hypothetical syllogism 1_, _generalized modus ponens_ and _generalized contraposition_ schemes (\(r=-1.0,p=0.000\)). In the FS setting (Fig. 15), the Gemma-7b-it model consistently exhibited significant decreases in _reasoning accuracy_ across all schemes, with the most pronounced effect in _hypothetical syllogism 3_ and _generalized modus tollens_ (\(r=-1.0p<0.000\)). Interestingly, the Mistral-7B Instruct model in both settings, depending on the scheme, showed a positive or negative significant correlation.

The findings underscore the substantial impact of distractors on reasoning accuracy, particularly in complex syllogistic reasoning tasks, revealing that current LLMs are highly susceptible to performance degradation as distractor complexity increases 

Results: Models Prioritize Contextual Knowledge Over Background Knowledge

The lack of statistically significant differences (Fig. 7) in accuracy between biologically factual and artificial datasets across _generalized modus ponens_ and _generalized modus tollens_ schemes suggests that the models' reasoning capabilities rely more on stated contextual knowledge and logical structure than on pre-existing background knowledge. This holds true for both _accuracy_ and _reasoning accuracy_, as well as in both ZS and FS settings: models that perform well on a given scheme maintain their performance even when factual gene names are replaced by synthetic names, and the same consistency is observed for models with weaker performance. This ability to maintain accuracy with synthetic gene names in the artificial set demonstrates that models can abstract and apply logical reasoning independently of their internal domain-specific knowledge.

## Appendix K Supplementary Figures
 

Figure 8: Percentage distribution of model response types under zero-shot settings for prompts with no distractors for the set of biologically factual argumentative texts.

Figure 7: Accuracy comparison between two datasets: Biologically Factual vs. Artificial with Synthetic Names for Task 1 (top) and Task 2 (bottom); ZS (left) and FS (right). Lines connect the accuracy for each model, with green indicating an increase and red indicating a decrease. Gray boxplots display the median, Q1, Q3, and the range (minimum to maximum) of the data.

Figure 9: Percentage distribution of model response types under few-shot settings for prompts with no distractors for the set of biologically factual argumentative texts.

 

Figure 12: Task 1: Accuracy versus number of distractors and scheme in the zero-shot setting. Lines connect the average values for each model, with error bars representing the range (min-max).

Figure 10: Percentage distribution of model response types under zero-shot settings for prompts with all distractors for the set of biologically factual argumentative texts.

Figure 11: Percentage distribution of model response types under few-shot settings for prompts with all distractors for the set of biologically factual argumentative texts.

 

* [15] M. C. B. Anderson, A. J. Barabasi, and R. S. Baker. The performance of the two-stage system in a multi-task setting setting. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 2078-2089, 2013.
* [16] D. P. Kingma and J. Ba. Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam: Adam Adam: Adam: Adam: Adam: Adam: Adam: Adam Adam: Adam: Adam: Adam: Adam Adam: Adam: Adam: Adam: Adam Adam: Adam: Adam: Adam Adam: Adam: Adam: Adam Adam: Adam: Adam: Adam Adam: Adam: Adam Adam: Adam: Adam Adam: Adam: Adam Adam: Adam: Adam Adam: Adam: Adam Adam: Adam: Adam Adam: Adam Adam: Adam: Adam Adam: Adam Adam: Adam: Adam Adam: Adam Adam: Adam Adam: Adam Adam: Adam Adam: Adam Adam: Adam Adam: Adam Adam: Adam Adam: Adam Adam: Adam Adam: Adam Adam: Adam Adam: Adam Adam: Adam Adam: Adam Adam: Adam Adam: Adam Adam: Adam Adam: Adam Adam: Adam Adam: Adam Adam: Adam Adam Adam: Adam Adam: Adam Adam Adam: Adam Adam Adam: Adam Adam Adam: Adam Adam Adam: Adam Adam Adam: Adam Adam Adam: Adam Adam Adam Adam: Adam Adam Adam Adam: Adam Adam Adam Adam Adam: Adam 



\begin{table}
\begin{tabular}{l l c r r r r r r r} \hline \hline Scheme & \multicolumn{1}{c}{**Model**} & \multicolumn{3}{c}{**Recall**} & \multicolumn{2}{c}{**Provision**} & \multicolumn{2}{c}{**F1-curve**} & \multicolumn{2}{c}{**Accuracy**} \\ \cline{5-8}  & **Zs** & **R’s** & **Zs** & **R's** & **Zs** & **R’s** & **Zs** & **R’s** & **R’s** \\ \hline \multirow{9}{*}{_disjunctive_**} & All models & 0.11 & 0.24 & 0.21 & 0.12 & 0.11 & 0.25 & 0.12 & 0.27 \\  & BioMariantTBD & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 \\  & Meta-Linear-3.8B-Inness & 0.01 & 0.45 & 0.13 & 0.03 & 0.38 & 0.01 & 0.02 \\  & Meta-Linear-3.8B-Inness & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 \\  & Meta-Linear-3.6B-Inness & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 \\  & Meta-Linear-3.0D-inness & 0.01 & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 \\  & Normal-3.6B-Inness+0.1 & 0.11 & 0.14 & 0.15 & 0.08 & 0.19 & 0.39 & 0.39 \\  & Germm-7b & 0.05 & 1.00 & 0.41 & 0.56 & 0.50 & 0.72 & **0.35** & 0.61 \\  & Commn-7a & 0.08 & 0.00 & 0.13 & 0.34 & 0.30 & 0.38 & 0.42 \\  & All models & 0.39 & 0.38 & 0.33 & 0.43 & 0.35 & 0.38 & 0.31 \\  & BiMoMariantTBD & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 \\  & Meta-linear-3.8B-Inness & 0.14 & 0.46 & 0.13 & 0.30 & 0.39 & 0.01 & 0.29 \\
**generalized** & Meta-Linear-3.8B-Inness & 0.10 & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 \\  & Neural-3.7B-Inness+0.1 & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 \\  & Meta-Linear-3.8B-Inness+0.1 & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 \\  & Normal-3.7B-Inness+0.1 & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 \\  & Memory-7a & 0.07 & 0.09 & 0.16 & 0.45 & 0.41 & 0.67 & 0.45 \\  & Modari TBD-nontest+0.1 & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 \\  & Maxit TBD-nontest+0.1 & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 \\  & German-7b & 0.97 & 0.61 & 0.41 & 0.55 & 0.47 & 0.49 & 0.40 \\  & Normal-3.6B-Inness+0.1 & 0.61 & 0.45 & 0.52 & 0.55 & 0.67 & 0.49 & 0.54 \\  & Gramm-7b & 0.2 & 0.10 & 0.52 & 0.50 & 0.36 & 0.46 & 0.52 \\  & German-7b & 0.97 & 0.61 & 0.67 & 0.75 & 0.73 & 0.68 & **0.27** \\ \hline \multirow{9}{*}{_generalized_**} & All models & 0.37 & 0.48 & 0.32 & 0.41 & 0.34 & 0.35 & 0.28 & 0.30 \\  & BioMariantTBD & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 \\  & Meta-Linear-3.8B-Inness & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 \\
**generalized** & Meta-Linear-3.8B-Inness+0.1 & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 \\
**dilemma** & Meta-Linear-3.8B-Inness+0.1 & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 \\  & Massit TBD-nontest+0.1 & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 \\  & Monat K-Inness+0.1 & 0.48 & 0.58 & 0.58 & 0.58 & 0.42 & 0.56 & 0.54 \\  & Gromm-7b & 0.79 & 0.40 & 0.53 & 0.38 & 0.42 & 0.56 & 0.54 \\  & Common-7b & 0.42 & 0.79 & 0.59 & 0.49 & 0.46 & 0.63 & **0.55** \\ \hline \multirow{9}{*}{_generalized_**} & All models & 0.51 & 0.50 & 0.43 & 0.45 & 0.47 & 0.41 & 0.44 \\  & BioMariantTBD & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 \\  & Meta-Linear-3.8B-Inness+0.0 & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 \\  & MediTBD-nontest+0.1 & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 \\
**generalized** & Meta-Linear-3.8B-Inness+0.0 & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 \\  & Mean-7b & 0.97 & 0.10 & 0.56 & 0.63 & 0.69 & 0.78 & 0.59 & 0.71 \\  & Gramm-7b & 0.74 & 0.37 & 0.66 & 0.67 & 0.71 & 0.70 & 0.70 & 0.70 \\ \hline \multirow{9}{*}{_generalized_} & All models & 0.00 & 0.28 & 0.34 & 0.28 & 0.36 & 0.28 & 0.31 & 0.29 \\  & BioMariantTBD & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 \\  & MedizTBD & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 \\  & Meta-Linear-3.8B-Inness+0.1 & 0.18 & 0.51 & 0.00 & 0.01 & 0.41 & 0.41 & 0.40 & 0.37 \\
**generalized** & Meta-Linear-3.8B-Inness+0.2 & 0.32 & 0.20 & 0.26 & 0.02 & 0.02 & 0.02 & 0.02 & 0.00 \\  & Normit TBD-nontest+0.2 & 0.32 & 0.32 & 0.47 & 0.49 & 0.48 & 0.35 & 0.59 & 0.89 \\  & MainTBD-nontest+0.1 & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 \\  & Market TBD-nontest+0.1 & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 \\  & Moment TBD-nontest+0.1 & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 \\  & Moment TBD-nontest+0.1 & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 \\  & Moment TBD-nontest+0.2 & 0.32 & 0.47 & 0.49 & 0.48 & 0.45 & 0.45 & 0.59 & 0.89 \\  & Moment TBD-nontest+0.1 & 0.00



\begin{table}
\begin{tabular}{l l c c c c c c c} \hline \hline Scheme & \multicolumn{2}{c}{Gamma-7b} & \multicolumn{3}{c}{Gamma-7b-it} & \multicolumn{3}{c}{Mistral-7ß} & \multicolumn{1}{c}{Mistral-8x7B} \\  & – & & & & Instruct-v0.2 & \multicolumn{1}{c}{Instruct-v0.1} \\ \cline{4-9}  & r & p-val & r & p-val & r & p-val & r & p-val \\ \hline ZS & & & & & & & & & \\ disjunctive syllogism & **-0.943** & **0.005** & 0.029 & 0.957 & -0.086 & 0.872 & 0.486 & 0.329 \\ generalized contraposition & **-0.943** & **0.005** & **-1.000** & **0.000** & **-0.829** & **0.042** & 0.429 & 0.397 \\ generalized dilemma & **-1.000** & **0.000** & -0.086 & 0.872 & **1.000** & **0.000** & 0.314 & 0.544 \\ generalized moths ponens & **-0.943** & **0.005** & **-1.000** & **0.000** & 0.543 & 0.266 & 0.771 & 0.072 \\ generalized moths tollens & **-0.943** & **0.005** & **-0.829** & **0.042** & **0.829** & **0.042** & 0.771 & 0.072 \\ hypothetical syllogism 1 hypothesis hypothesis & **-1.000** & **0.000** & **-1.000** & **0.000** & 0.029 & 0.957 & -0.486 & 0.329 \\ syllogism 3 & **-0.886** & **0.019** & **-0.943** & **0.005** & **-0.943** & **0.005** & 0.029 & 0.957 \\ \hline FS & & & & & & & & \\ disjunctive syllogism & **-1.000** & **0.000** & -0.543 & 0.266 & **0.829** & **0.042** & -0.143 & 0.787 \\ generalized contraposition & -0.086 & 0.872 & **-0.943** & **0.005** & **-0.943** & **0.005** & **0.829** & **0.042** \\ generalized dilemma & -0.657 & 0.156 & **-0.943** & **0.005** & **-0.829** & **0.042** & **0.943** & **0.005** \\ generalized modus ponens & 0.371 & 0.468 & **-0.886** & **0.019** & 0.600 & 0.208 & **0.829** & **0.042** \\ generalized mouts tollens & 0.486 & 0.329 & **-1.000** & **0.000** & 0.371 & 0.468 & 0.257 & 0.623 \\ hypothetical syllogism 1 hypothesis & -0.086 & 0.872 & **-0.943** & **0.005** & 0.314 & 0.544 & **0.829** & **0.042** \\ hypothetical syllogism 3 & **-1.000** & **0.000** & **-1.000** & **0.000** & **-0.943** & **0.005** & -0.486 & 0.329 \\ \hline \end{tabular}
\end{table}
Table 8: Values of the Spearman’s Ranked Correlation Coefficient (r) for Accuracy by Distractors and Syllogistic Scheme for evaluated Models in Task 2: Spearman’s correlation coefficients (r) and p-values for the reasoning accuracy metric are shown across various levels of distractors and syllogistic schemes for each model. Negative r values reflect a decrease in accuracy with increasing distractor complexity. The highest correlation values for each scheme are highlighted in bold, indicating the models most affected by distractors.