<figure><img src="image_1.png" /></figure>

HAL
open science

# Investigating Language Model Capabilities to Represent and Process Formal Knowledge: A Preliminary Study to Assist Ontology Engineering

Hanna Abi Akl

► To cite this version:

Hanna Abi Akl. Investigating Language Model Capabilities to Represent and Process Formal Knowledge: A Preliminary Study to Assist Ontology Engineering. the International Joint Conference on Rules and Reasoning (RuleML+RR) 2025, Sep 2025, Instabul, Turkey. hal-05248053v2

HAL Id: hal-05248053

https://inria.hal.science/hal-05248053v2

Submitted on 11 Sep 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Investigating Language Model Capabilities to Represent and Process Formal Knowledge: A Preliminary Study to Assist Ontology Engineering

Hanna Abi Akl¹

¹Supervised by Fabien Gandon, Catherine Faron and Pierre Monnin
Université Côte d'Azur, Inria, CNRS, I3S, Sophia Antipolis, France and Data ScienceTech Institute (DSTI), Paris, France

**Abstract**

Recent advances in Language Models (LMs) have failed to mask their shortcomings particularly in the domain of reasoning. This limitation impacts several tasks, most notably those involving ontology engineering. As part of a PhD research, we investigate the consequences of incorporating formal methods on the performance of Small Language Models (SLMs) on reasoning tasks. Specifically, we aim to orient our work toward using SLMs to bootstrap ontology construction and set up a series of preliminary experiments to determine the impact of expressing logical problems with different grammars on the performance of SLMs on a predefined reasoning task. Our findings show that it is possible to substitute Natural Language (NL) with a more compact logical language while maintaining a strong performance on reasoning tasks and hope to use these results to further refine the role of SLMs in ontology engineering.

**Keywords**

Small Language Models, Logical Reasoning, Knowledge Representation

## 1. Introduction

Since their introduction, Language Models (LMs) have achieved impressive results on a broad spectrum of Natural Language Processing (NLP) tasks, ranging from text classification to question-answering and text generation [1, 2]. One field that still poses a challenge for LMs is reasoning. Over different kinds of reasoning tasks, LMs still struggle to achieve comparable results to humans in tasks involving explicit or implicit thinking (e.g. mathematical, commonsense, logical first-order reasoning) [1, 3]. These reasoning capabilities are essential for complex tasks such as knowledge graph construction or ontology engineering [4, 5]. For the latter, LMs have been used in various roles in the ontology learning pipeline, from requirement gathering to ontology creation, validation and alignment [6, 7, 8, 9, 10]. Current research promotes neuro-symbolic approaches integrating symbolic mechanisms to curb LMs as current fine-tuning techniques (e.g. prompting) have proven insufficient in making them reliable reasoners [11, 12].

In this perspective, the goal of our research in this PhD is to evaluate reasoning in LMs by focusing on small language models (SLMs) for the frugality, autonomy and privacy they support and by narrowing our evaluation to first-order logic (FOL) reasoning. We present and evaluate an experimental setup for testing the performance of SLMs on classifying the satisfiability of a logical conclusion given a set of premises. We also experiment with different knowledge representation languages to measure the impact of the formal representation of the input on the performance of SLMs on reasoning tasks. In doing so, we aim to address the following research question (RQ): *RQ: Is there a better formal representation for logical data than natural language?*

This paper is structured as follows: Section 2 provides a review of the state of the art in LM reasoning methods in the context of formal problem resolution with a focus on logic tasks. Section 3 introduces

RuleML+RR'25: Companion Proceedings of the 9th International Joint Conference on Rules and Reasoning, September 22-24, 2025, Istanbul, Turkeye

hanna.abi-akl@inria.fr (H. A. Akl)

https://hannaabiakl.github.io/ (H. A. Akl)

0000-0001-9829-7401 (H. A. Akl)

<figure><img src="image_3.png" /></figure>

© 2025 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
our methodology and formalizes the Common Logic Grammar Construction (CLGC) pipeline. Section 4 presents our extended experimental setup and reports our findings with discussions on our results. Finally, Section 5 provides a summary of our key findings and outlines the next steps in this PhD research as well as potential future research directions.

## 2. State of the Art

The focus of the first stage of this PhD is to combine formal knowledge representation with LM learning methods to improve LM performance on FOL reasoning tasks with the perspective of an application to ontology engineering. We report here the relevant research regarding state-of-the-art LM reasoning methods and logical reasoning applications.

### 2.1. Language Model Reasoning

Reasoning is a fundamental aspect of human intelligence and is a cognitive process that involves the use of evidence, arguments and logic to arrive at conclusions [13]. The impressive performance of LMs on Natural Language (NL) data has led to observations that these models may exhibit reasoning abilities when they are sufficiently large [13]. However, reasoning also appears to be inherent in SLMs which are shown to be able to perform close to, and in some cases on par with, Large Language Models (LLMs). Approaches include extensive pre-training or guided techniques such as problem decomposition or the introduction of LLMs in SLM training processes for step correction as in the case of the SMART framework [14, 15, 16].

In an effort to determine reasoning abilities in LMs, researchers have explored novelties in techniques, benchmark datasets and evaluation metrics [13]. Reasoning techniques largely fall under two pillars: LM-based and Reinforcement Learning (RL) techniques [17]. LM-based reasoning techniques revolve around the methodologies spearheading model training, i.e. Supervised Fine-Tuning (SFT) on the one hand and Prompting and In-Context Learning on the other [13, 18]. Another emerging research direction is the exploration of hybrid or neuro-symbolic models which incorporate a form of symbolic knowledge into LMs to guide or enhance reasoning as seen in DomiKnowS and Logic [19, 20]. Different training techniques extending fine-tuning and prompting like COCONUT have resulted in LMs being used to tackle various reasoning tasks like mathematical, logical, causal, commonsense reasoning and coding [21]. None of these methods however consider the potential impact of the logical formalism used to prompt the LMs.

### 2.2. Logical Reasoning Tasks for LMs

Logical reasoning is an interesting domain of application for LMs since it subjects them to different types of reasoning, namely deductive, inductive, abductive or analogical [13]. Reasoning problems are classified in the literature into 2 main groups: Logical Question Answering (LQA) and Logical Consistency (LC) [22]. LQA applications encompass tasks that require LMs to generate the correct answer within complex logical problems which require sophisticated reasoning given a collection of premises and constraints [22]. LC applications require LMs to generate verifiable answers to complex questions without violating consistency (e.g. the model should not generate answers that contain contradictions) [22]. As we will show, our first research questions, methods and experimental setup fall under LQA and rely on FOLIO, a human-annotated benchmark dataset for FOL reasoning [23].

LQA models can be grouped into 3 categories depending on the approach. Solver-aided models rely on LMs translating NL problems into FOL before passing them to a solver to compute the final answer. These models have shown an increased performance over traditional model training techniques on logical problem-solving [24, 25]. Prompt-based models rely on techniques like Chain-of-Thought (CoT) to break down complex problems using smaller reasoning steps (i.e. hops) and guide the model to the final solution [26]. However, experiments on these models did not prove actual reasoning beyond two hops and could not determine that this type of reasoning scales with larger models [26, 27]. The last
group of models adhere to the pre-training and fine-tuning approaches on logical questions to enhance their reasoning process. These models often include an additional neural layer that captures logical constraints and benefits from forward and backward passes to iteratively refine the model output in the constrained setting [28]. Our methodology targets the data representation of the input and evaluates models with different representations in both the prompting and fine-tuning settings. Research on input representation has already shown that the language in which a LM receives a problem has an impact on its success rate in solving the problem [29], but does not consider the variety of languages we evaluate in our work.

# 3. SEF-CLGC Methodology

This section introduces our methodology in three steps: (1) we characterize and categorize the different syllogisms that we will confront the LMs with, then (2) we introduce the different grammars we will use and evaluate in solving the logical reasoning tasks and (3) we provide an overview of the pipeline implementing the complete experiment. The following sub-sections present the Syllogistic Evaluation Framework (SEF) and Common Logic Grammar Construction (CLGC) pipeline which are integral components for studying the impact of different formal representations on reasoning tasks as part of this PhD research. The last sub-section illustrates the process to derive the CLGC pipeline as well as the generated grammars which constitute the crux of our experimental studies.

## 3.1. Syllogistic Evaluation Framework to Characterize the Logical Reasoning Tasks

The Syllogistic Evaluation Framework (SEF) is a methodology for identifying and classifying different types of syllogisms inspired by the work in [30]. The objective of SEF is to have an additional evaluation criterion to trace and analyze performance on different types of logical reasoning: SEF allows discrimination between different reasoning problems to explicit a model's reasoning capabilities. We applied SEF on the FOLIO dataset which is the focal data source for our experiments. FOLIO is a collection of stories, with each story *S* consisting of a set of *n* premises *P* = {*p*₁, *p*₂, ..., *p*ₙ} and *m* conclusions *H* = {*h*₁, *h*₂, ..., *h*ₘ} in NL and their corresponding FOL annotations of *n* premises *PF* = {*p*f₁, *p*f₂, ..., *pfₙ} and *m* conclusions *HF* = {*hf*₁, *hf*₂, ..., *hf*ₘ}. Given each premises-conclusions pair, the goal is to determine the truth values of the conclusions, i.e. "True", "False" or "Uncertain" based on FOL reasoning. We determined 4 syllogistic categories to classify each premises-conclusions pair: Disjunctive, Hypothetical, Categorical and Complex. The categories were identified based on the following logical criteria presented from highest to lowest order of precedence: (1) Disjunctive: Any premises-conclusions pair containing a disjunction ∨ or ⊕ (2) Hypothetical: Any premises-conclusions pair containing an implication = ⇒ (3) Categorical: Any premises-conclusions pair consisting of exactly 2 premises and a conclusion and (4) Complex: The default category for premises-conclusions pairs that do not fit in any of the three other categories.

Examples of SEF classification are presented in Table 1. The classification statistics for FOLIO are presented in Table 2. The distribution shows an imbalance heavily favoring Disjunctive and Hypothetical kinds of reasoning, while Complex and Categorical examples are too sparsely represented to draw any conclusions on.

## 3.2. Alternative Languages for Transcribing the Logical Reasoning Tasks

Different knowledge representation formalisms have been tested in the CLGC pipeline to measure the impact of the language on the SLM in solving a logical reasoning task. We present concrete transformations from FOL to CLGC languages in Table 3.

### 3.2.1. Common Logic

We implemented grammars from the Common Logic (CL) family suite as defined in [31, 32, 33]. In particular, we focused on the Common Logic Interchange Format (CLIF) and the Conceptual Graph
<table><caption>**Table 1**
SEF classification examples on FOLIO.</caption><thead><tr><th>NL Premise</th><th>NL Conclusion</th><th>FOL Premise</th><th>FOL Conclusion</th><th>SEF Class</th></tr></thead><tbody><tr><td>All squares are four-sided. All four-sided things are shapes.</td><td>All squares are shapes.</td><td>∀x (Square(x) → FourSided(x))<br/>∀x (FourSided(x) → Shape(x))</td><td>∀x (Square(x) → Shape(x))</td><td>Hypothetical</td></tr><tr><td>Some affection is love. Some love is positive.</td><td>Some affection is positive.</td><td>∃x (Affection(x) ∧ Love(x))<br/>∃x (Love(x) ∧ Positive(x))</td><td>∃x (Affection(x) ∧ Positive(x))</td><td>Categorical</td></tr><tr><td>Diamond Mine is a professional wrestling stable formed in WWE. Roderick Strong leads Diamond Mine. Diamond Mine includes the Creed Brothers and Ivy Nile. Imperium has a feud with Diamond Mine.</td><td>Roderick Strong leads the Creed Brothers.</td><td>ProfessionalWrestlingStable(diamondMine) ∧ In(diamondMine, WWE)<br/>Leads(roderickStrong, diamondMine)<br/>Includes(diamondMine, creedBrothers) ∧ Includes(diamondMine, ivyNile)<br/>Feuds(imperium, diamondMine)</td><td>Leads(roderickStrong, creedBrothers)</td><td>Complex</td></tr><tr><td>Susan flies to LGA airport. The departure and arrival can not be at the same airport. John flies from LGA airport.</td><td>Susan flies from LGA airport.</td><td>FlyTo(susan, IgaAirport)<br/>∀x ∀y (FlyFrom(x, y) ⊕ FlyTo(x, y))<br/>FlyFrom(john, IgaAirport)</td><td>FlyFrom(susan, IgaAirport)</td><td>Disjunctive</td></tr></tbody></table>

<table><caption>**Table 2**
Syllogistic Evaluation Framework Statistics on FOLIO.</caption><thead><tr><th>Syllogism</th><th>Train</th><th>Test</th><th>Valid</th></tr></thead><tbody><tr><td>Disjunctive</td><td>414</td><td>113</td><td>99</td></tr><tr><td>Hypothetical</td><td>285</td><td>68</td><td>84</td></tr><tr><td>Complex</td><td>114</td><td>22</td><td>16</td></tr><tr><td>Categorical</td><td>6</td><td>2</td><td>4</td></tr></tbody></table>

Interchange Format (CGIF) languages for their ease of interoperability with FOL while retaining the advantages of formal logic representations. From a practical aspect, these languages can also be implemented due to the availability of their grammars<fnref n="1" />. We limited our implementation to the core Backus-Naur form (BNF) grammars for CLIF and CGIF and did not include the extended syntax for these languages for simplicity and compactness.

### 3.2.2. Tensor Function Logic and Tensor Function Logic Plus

We also incorporated Tensor Function Logic (TFL), a formal reasoning language introduced in [34, 35, 36], and its extension, Tensor Function Logic Plus (TFL+). We limited the scope of TFL to the terms and their arithmetic signs and incorporated quantifier subscript representation as well as parentheses in TFL+ to work with grammars of varying complexity.

### 3.2.3. MINIFOL

Additionally, we implemented a custom language called Miniature First-Order Logic (MINIFOL) which is directly derived from FOL by replacing $∀$ and $∃$ with “all” and “some” respectively and replacing operators with Boolean equivalents (e.g. “∧” with “&”). The advantage of MINIFOL is twofold: replace some FOL vocabulary (i.e. symbols) with vocabulary that is more familiar for SLMs (i.e. words) while benefiting from a rule set that enables an easy transformation from FOL to MINIFOL.

<footnote n="1">¹https://www.w3.org/DesignIssues/Sowa/cgstand.htm</footnote>
<table><caption>**Table 3**
Example transformations from FOL to CLGC languages.</caption><thead><tr><th>FOL</th><th>MINIFOL</th><th>CLIF</th><th>CGIF</th><th>TFL</th><th>TFL+</th></tr></thead><tbody><tr><td>∀x ((Employee(x) ∧ Schedule(x, meeting, customers)) → AppearIn(x, company))</td><td>all:x ((employee(x) & schedule(x, meeting, customers)) :- appearin(x, company))</td><td>forall x ((employee(x) and schedule(x, meeting, customers)) implies appearin(x, company))</td><td>[@every *x [[((employee[(?x)] schedule[(?x] meeting customers)))] appearin[(?x company)]]]</td><td>--E1++S1+A1</td><td>- ((+E0++S0)-+A0)</td></tr><tr><td>∀x ((Employee(x) ∧ HasLunch(x, company)) → Schedule(x, meeting, customers))</td><td>all:x ((employee(x) & haslunch(x, company)) :- schedule(x, meeting, customers))</td><td>forall x ((employee(x) and haslunch(x, company)) implies schedule(x, meeting, customers))</td><td>@every *x [[((employee[(?x)] haslunch[(?x company)] schedule[(?x meeting customers)]])]]</td><td>--E1++H1+S1</td><td>- ((+E0++H0)-+S0)</td></tr><tr><td>∀x (Employee(x) → (HasLunch(x, company) ⊕ HasLunch(x, home)))</td><td>all:x (employee(x) :- (haslunch(x, company) haslunch(x, home)))</td><td>forall x (employee(x) implies (haslunch(x, company) xor haslunch(x, home)))</td><td>@every *x [[((employee[(?x)] (haslunch[(?x company)] haslunch[(?x home)]]))]]</td><td>--E1+-H1+H1</td><td>- (+E0-(-H0+H0))</td></tr><tr><td>∀x ((Employee(x) ∧ HasLunch(x, home)) → Work(x, home))</td><td>all:x ((employee(x) & haslunch(x, home)) :- work(x, home))</td><td>forall x ((employee(x) and haslunch(x, home)) implies work(x, home))</td><td>@every *x [[((employee[(?x)] haslunch[(?x home)]))]] work[(?x home)]]]</td><td>--E1++H1+W1</td><td>- ((+E0++H0)-+W0)</td></tr><tr><td>∀x ((Employee(x) ∧ (¬In(x, homecountry)) → Work(x, home))</td><td>all:x ((employee(x) & (in(x, homecountry)))-: work(x, home))</td><td>forall x ((employee(x) and (not in(x, homecountry)) implies work(x, home))</td><td>@every *x [[((employee[(?x)] [(in[(?x homecountry)))] work[(?x home)]]))]]</td><td>--E1+-+I1+W1</td><td>- ((+E0+(-+I0))-+W0)</td></tr><tr><td>∀x (Manager(x) → ¬Work(x, home))</td><td>all:x (manager(x) :- work(x, home))</td><td>forall x (manager(x) implies not work(x, home))</td><td>@every *x [[(manager[(?x)] work[(?x home)]])]</td><td>- +M1-+W1</td><td>- (+M0-+W0)</td></tr><tr><td>¬(Manager(james) ⊕ AppearIn(james, company))</td><td>(manager(james) ∧ appearin(james, company))</td><td>not (manager(james) xor appearin(james, company))</td><td>[(manager[(james)] appearin[(james company)]])]</td><td>--M1-+A1</td><td>- (+M2(+j2)-+A2)</td></tr><tr><td>HasLunch(james, company)</td><td>haslunch(james, company)</td><td>haslunch(james, company)</td><td>[haslunch[(james company)]]]</td><td>+H1</td><td>+H2</td></tr></tbody></table>

### 3.3. CLGC Pipeline and Alternative Configurations

Because this work investigates the impact of the formal representation of knowledge in the scope of reasoning tasks, we devised a pipeline to transform one logical form to another with the goal of generating different data representations from FOLIO to test as input to our SLMs. The CLGC pipeline is presented in Figure 1 and consists of the following steps: (1) FOLIO input data (i.e. a premises-conclusions pair) is passed along with an input grammar (i.e. FOL) in BNF to a parser. We used the Python Lark<fnref n="2" /> parser in our work. (2) The parser generates an input grammar parse tree for the input language. Figure 2 shows a concrete example of generating the corresponding tree for a FOL input. (3) The parse tree is passed to an input-output grammar function which generates a new parse tree in the desired language (e.g. CLIF). The algorithm for the transformation is detailed in Algorithm 1. The implementation relies heavily on the BNF grammar definition of the input and target languages as it uses the non-terminal symbol definitions of both grammars to construct the target parse tree from the input tree by identifying the corresponding mappings and augmenting the target tree recursively. (4) The algorithm either successfully generates the parse tree in the target language or returns an error. In the event of the latter, the input and target BNF grammars are reviewed and corrected manually. (5) The target parse tree is passed along with its corresponding language BNF grammar to the Lark parser. (6) The parser generates an output string corresponding to the output data (i.e. premises-conclusions pair) in the target language. The output is passed to a formatting function to align the syntax with the target language (e.g. spacing). (7) The pipeline outputs a correctly-formatted premises-conclusions pair in the target language. Figure 2 summarizes Steps 5-7.

<footnote n="2">²https://lark-pARSER.readthedocs.io/en/stable/</footnote>
<figure><img src="image_4.png" /><figcaption>Figure 1: CLGC pipeline.</figcaption></figure>

<figure><img src="image_5.png" /><figcaption>**Figure 2:** (Left) Input-to-tree example for Steps 1 and 2 of the CLGC pipeline. (Right) Tree-to-output example for Steps 5, 6 and 7 of the CLGC pipeline. The text in red represents the grammar of choice for the example.</figcaption></figure>

Additionally, we introduced 2 new SFT configurations: Grammar Context-Passing (ICGP) and Tokenizer Re-Training. In ICGP, the BNF grammar of the input data language is passed as additional context to the classical SFT input (i.e. the logical premises, conclusions and label). In the Tokenizer Re-Training setup, the SLM tokenizer is retrained on the vocabulary of the grammar by passing all train, test and validation data in the specified language to the model. After re-training, the original size of the tokenizer is kept in some runs to incorporate the vocabulary of the grammar and sufficiently resized empirically in others to eliminate as many tokens outside those making up the target vocabulary.

# 4. Experiments and Results

In this section, we describe our experimental setup and set out to answer the following research sub-questions (SRQs): SRQ1: Which training method yields the best results for solving FOL problems with LMs? SRQ2: How do formal representations scale with models? SRQ3: Does having a more compact vocabulary
**Algorithm 1** Input-Output Grammar Transformation Algorithm

1:
**Require:** $G_i := L_i$
$D_i \in \{L_i\}$
$D_o \in \{L_o\}$
$G_o := L_o$
2:
**Ensure:** $|L_i| < \infty$
$|L_o| < \infty$
$D_i \neq \emptyset$
$D_o = \emptyset$
3: **for** $d \in D$ **do**
4: &nbsp;&nbsp;&nbsp;&nbsp; $P_i \leftarrow generateParseTree(d, G_i)$
5: &nbsp;&nbsp;&nbsp;&nbsp; $P_o \leftarrow \emptyset$
6: &nbsp;&nbsp;&nbsp;&nbsp; ▷ generate grammar parse tree from statement
6: &nbsp;&nbsp;&nbsp;&nbsp; ▷ initialize output grammar parse tree for statement
7: &nbsp;&nbsp;&nbsp;&nbsp; **for** $p \in P_i$ **do**
8: &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; $o \leftarrow mapToParseTree(p, G_o)$
8: &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; ▷ loop through each element of the input grammar parse tree
8: &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; ▷ map input element to corresponding output grammar element
9: &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; **if** $o \neq \emptyset$ **then**
10: &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; $P_o \leftarrow o$
11: &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; **else**
12: &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; continue
13: &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; **end if**
14: &nbsp;&nbsp;&nbsp;&nbsp; **end for**
15: &nbsp;&nbsp;&nbsp;&nbsp; $v \leftarrow validateParseTree(P_o, G_o)$
16: &nbsp;&nbsp;&nbsp;&nbsp; **if** $v = True$ **then**
17: &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; $D_o \leftarrow P_o$
18: &nbsp;&nbsp;&nbsp;&nbsp; **else**
19: &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; $G_o \leftarrow updateGrammar(P_o, G_o)$
20: &nbsp;&nbsp;&nbsp;&nbsp; **end if**
21: **end for**

$\triangleright$ loop through logical statements

$\triangleright$ loop through each element of the input grammar parse tree

$\triangleright$ add mapped element to output grammar parse tree

$\triangleright$ check if final parse tree from statement is parsable

$\triangleright$ update output grammar to parse mapped parse tree

boost model performance for formal languages?

4.1. Experimental Setup

We extend the experiments performed in [23] on the NL reasoning with FOL task by expanding the
data representation of the premises and conclusions as well as the choice of models. For the data
representation, we retain the NL and FOL languages in addition to the ones we derived from our
pipeline in section 3 (i.e. CLIF, CGIF, TFL, TFL+ and MINIFOL). For our model selection, we restricted
our choices to SLMs for their frugality and promising performances on the original task in [23] and
because they avoid having a dependency to external and possibly expensive LLM services. We configure
our models (M), grammars (G) and learning methods (L) with the values: M = {Flan-T5-small, Flan-
T5-base, Flan-T5-large, GPT-2, Phi-3.5-mini-instruct, Gemma-2-2b-it}, G = {NL, FOL, CLIF, CGIF, TFL,
TFL+, MINIFOL} and L = {SFT, Zero-Shot (ZS) Prompting, Few-Shot (FS) Prompting}.

Our experimental runs consisted of variations of the triple (M, G, L) to determine the truth value of a
conclusion (i.e. “True”, “False” or “Uncertain”) based on FOL reasoning over a set of premises. We split
the provided train portion of the original version of FOLIO<fnref n="3" /> into 80% train and 20% test and froze those
splits along with the original validation set for all our experimental runs. Our final splits consisted of
800 training, 203 validation and 201 test rows.

<footnote n="3">³https://huggingface.co/datasets/yale-nlp/FOLIO</footnote>
<table><caption>**Table 4**
Example templates of basic and BNF grammar prompts. The {...} are replaced with real data.</caption><thead><tr><th>Prompt Type</th><th>Example</th></tr></thead><tbody><tr><td>Basic</td><td>You are given a set of premises and a conclusion. The premises start after a &lt;PREMISES&gt; tag and conclusion starts after a &lt;CONCLUSION&gt; tag. Classify each conclusion as "True", "False" or "Uncertain" depending on its satisfiability of the premises. &lt;PREMISES&gt;...&lt;/PREMISES&gt; &lt;CONCLUSION&gt;...&lt;/CONCLUSION&gt;</td></tr><tr><td>BNF Grammar</td><td>You are an expert in logic, and you need to output the conclusion for the following logic problem. You are given grammar rules for the language of the problem in &lt;GRAMMAR&gt; tags, and you should conform to these rules to understand the premises in the &lt;PREMISES&gt; tags and conclusion in the &lt;CONCLUSION&gt; tags. Classify each conclusion as "True", "False" or "Uncertain" depending on its satisfiability of the premises. Present your answer only in &lt;output&gt; tags. &lt;GRAMMAR&gt;...&lt;/GRAMMAR&gt; &lt;PREMISES&gt;...&lt;/PREMISES&gt; &lt;CONCLUSION&gt;...&lt;/CONCLUSION&gt;</td></tr></tbody></table>

All experiments were run on Google Colab instances using L4 GPUs and, depending on availability, A100 High-RAM GPUs. In order to run some configurations in the absence of A100 GPUs (e.g. SFT with Flan-T5-large), we used Parameter-Efficient Fine-Tuning with Low-rank adaptation (PEFT-LoRA). PEFT-LoRA freezes the pre-trained model weights to reduce the number of trainable parameters and has shown great efficacy for LM training [37]. In our setup, PEFT-LoRA is configured as follows: `{r=16, lora_alpha=32, target_modules=["q", "v"], lora_dropout=0.05, bias=none`, `task_type="SEQ_2ikh_LM"`). In the SFT mode, all models were trained on 5 epochs without PEFT-LoRA and 10 epochs with PEFT-LoRA.

For the ZS and FS modes, we used 2 prompt templates. The first is a basic template explaining the task, the nature of the input and the expected output. The second is the basic template augmented with the BNF grammar rules of the language when applicable (i.e. for languages other than NL). We performed 8-shot prompting for all FS experiments. Table 4 shows an example of a prepared basic prompt and augmented prompt template for the ZS and FS experiments. No ablation study was performed on the CLGC pipeline as all components are needed to generate a correct grammar for our experiments.

## 4.2. Results

We present the most prominent results from our experimental runs in Tables 5-13. Table 5 shows that formalizing the premises and conclusions in CLIF ties the performance of NL on Accuracy and ranks second-best on F1 score. The result shows that SLMs can perform well on first-order reasoning tasks with a more compact formalism than NL. Languages with more complex syntaxes like CGIF struggle to yield good performances, suggesting that it is more difficult for SLMs to reason over complex representations. It is also worth noting that the best performances occur with the smallest model, Flan-T5-small, which outperforms larger, fine-tuned models (e.g. Flan-T5-large), suggesting that added architectural complexity may hinder the learning process in this setting. The result is empirically supported by the different data formalisms in our runs.

Table 6 shows scaled results on bigger models. With the A100 GPUs, models like Flan-T5-base and Flan-T5-large can be trained in supervised fashion without PEFT-LoRA tuning. The results confirm our earlier findings that a compact language like CLIF can effectively model logical data and be competitive on a reasoning task by ranking second-best to NL, i.e. the current state of the art in [23].

Table 7 shows the impact of languages in the ZS setting. The results show that the performance of compact languages like CLIF and TFL+ are extendable to other models, as shown with Gemma-2-2b-it which ranks first in Precision with TFL+ and yields competitive results with CLIF. Not all grammars are created equally however, as shown by MINIFOL and TFL which for very small models like Flan-T5-small cripple the learning completely resulting in a flat performance of 0.

The results of Table 8 cement CLIF as the best language choice in the ZS setting. With and without BNF grammar prompting, CLIF outperforms all other languages. In addition, grammar prompting
seems to improve SLM performance in the ZS case over first-order reasoning as empirically shown for all our languages. These results support similar findings in [38].

In Table 9 we present our results for the 8-shot setting. The performances are comparable to the ZS case in Table 7 and cement CLIF as a more compact contender to the verbose NL representation of logical input. A familiar pattern seen in the ZS case re-emerges as Flan-T5-small is unable to learn with TFL. This suggests a limitation to model learning with respect to size as even for very small grammars like TFL, very small models struggle to learn the formalism well enough to solve a reasoning task.

Table 10 showcases the effects of augmenting the FS prompt with the language BNF grammars. The first takeaway is that augmenting the basic prompt template with the BNF grammar does not seem to impact the performance of a model in the FS case, unlike the ZS setting in Table 8. As for performance, the language choice does not seem to affect the model in the ZS setting, suggesting a model in that case might be less sensitive to the input representation. Finally, comparing these results to those in Table 8 shows that the same model performs better in the ZS case than the FS case. One explanation could be that the number of FS examples is not sufficient for the chosen model on this particular task for learning. Another plausible hypothesis can be that the examples might be skewing the learning of the model by adding noise in the case of solving a reasoning task.

Table 11 shows the results of passing BNF grammars as context in addition to the standard input in the SFT setting. Empirical results confirm that including the BNF grammar in-context hinders learning and degrades model performance on the first-order reasoning task. Model size and complexity does not seem to impact performance as scaling models still yields better results without the inclusion of the grammar. Comparing results to those in Tables 5 and 6 also shows weaker performance with in-context information passing than traditional supervised training on inputs.

Table 12 shows the impact of re-training a model tokenizer to adapt to the specific vocabulary of a grammar and re-using the tokenizer in the SFT setting. The results show promise in the case of smaller models for more compact data representations. This is illustrated by the performance of Flan-T5-small on TFL+ with tokenizer re-training and vocabulary resizing which outperforms Flan-T5-small on TFL+ without tokenizer re-training and resizing. This configuration even outperforms Flan-T5-small and Flan-T5-large with PEFT-LoRA on NL from Table 5, both of which represent the baseline of SFT on lower resources (i.e. L4 GPUs) for our task. However, as shown when moving to bigger models like Flan-T5-base and Flan-T5-large on TFL+, this method is not scalable and breaks down when tokenizer re-training is done, resulting in worse performances than using the default tokenizer for the models. One possible explanation is that re-training the tokenizer for small models and sufficiently compact data representations may make the models learn the formalism and solve the task more efficiently but also causes overfitting. This risk suggests that this method may not be adapted to take full advantage of the compactness of data representations as it sacrifices generalization and weakens overall performance on the reasoning task.

Finally, Table 13 uses SEF to showcase the predictive performances of our best configuration (i.e. Flan-T5-large in SFT setting) on the 3 best grammars (i.e. NL, CLIF and TFL+) on the FOLIO validation set. In all 3 cases, the model performs well on the Disjunctive and Hypothetical syllogisms which can be seen as an unsurprising result given the over-representation of these two types in FOLIO. Irrespective of the data representation, the model seems to also perform well on the Complex syllogisms, with TFL+ even yielding a slightly better performance on that type than NL and CLIF. Complex syllogisms typically contain 3 or more premises and polysyllogisms which might render them too ambiguous for NL and CLIF and may benefit more compact representations like TFL+. This may explain the slight increase in performance noted in the results. As for the Categorical syllogisms which are heavily under-represented, all grammars share equal performance with 2 hits and 2 misses on the prediction labels, but the data imbalance makes it difficult to interpret the results. Further investigation is needed on the quality of these predictions, especially in a comparative framework, to assess the capabilities of the model on different data representations. We provide an example of extended error analysis in Table 14. The analysis is conducted for the Categorical syllogistic type and shows that the variation in grammar generally results in the same model reasoning on these problems with the exception of TFL+ on a single occurence, whereby the model reasons "False" as opposed to "Uncertain" on NL and CLIF. A form of
<table><caption>**Table 5**
Supervised Fine-Tuning Results on L4 GPU: best results in bold and the second best underlined.</caption><thead><tr><th>Model</th><th>Grammar</th><th>Accuracy</th><th>Precision</th><th>Recall</th><th>F1</th></tr></thead><tbody><tr><td>Flan-T5-small</td><td><strong>NL</strong></td><td><strong>0.4384</strong></td><td><strong>0.4385</strong></td><td><strong>0.4355</strong></td><td><strong>0.4351</strong></td></tr><tr><td>Flan-T5-small</td><td><strong>CLIF</strong></td><td><strong>0.4384</strong></td><td><strong>0.4425</strong></td><td><strong>0.4274</strong></td><td><strong>0.4109</strong></td></tr><tr><td>Flan-T5-small</td><td>NL + FOL</td><td><u>0.4384</u></td><td>0.4078</td><td>0.4261</td><td>0.3951</td></tr><tr><td>Flan-T5-small</td><td>FOL</td><td>0.4236</td><td>0.3983</td><td>0.4108</td><td>0.3827</td></tr><tr><td>GPT-2*</td><td>NL</td><td>0.3743</td><td>0.3675</td><td>0.3713</td><td>0.3677</td></tr><tr><td>GPT-2*</td><td>NL + FOL</td><td>0.3793</td><td>0.3878</td><td>0.3844</td><td>0.3671</td></tr><tr><td>Flan-T5-small</td><td>CGIF</td><td>0.3743</td><td>0.3609</td><td>0.3666</td><td>0.3398</td></tr><tr><td>Flan-T5-large*</td><td><strong>NL</strong></td><td><strong>0.4729</strong></td><td>0.3391</td><td>0.3396</td><td>0.3391</td></tr><tr><td>Flan-T5-small</td><td>TFL</td><td>0.3596</td><td>0.3424</td><td>0.3476</td><td>0.3070</td></tr><tr><td>Flan-T5-large*</td><td>NL + FOL</td><td>0.4088</td><td>0.3566</td><td>0.2915</td><td>0.3017</td></tr><tr><td>Flan-T5-large*</td><td>TFL</td><td>0.4334</td><td>0.2858</td><td>0.3112</td><td>0.2962</td></tr><tr><td>Flan-T5-small</td><td>TFL+</td><td>0.3596</td><td>0.3243</td><td>0.3440</td><td>0.2918</td></tr><tr><td>GPT-2*</td><td>FOL</td><td>0.3349</td><td>0.3021</td><td>0.3260</td><td>0.2902</td></tr><tr><td>Flan-T5-large*</td><td>TFL+</td><td>0.2758</td><td>0.2326</td><td>0.1988</td><td>0.2143</td></tr><tr><td>Flan-T5-large*</td><td>FOL</td><td>0.3251</td><td>0.2096</td><td>0.2323</td><td>0.2086</td></tr><tr><td>Flan-T5-large*</td><td>CLIF</td><td>0.2709</td><td>0.2297</td><td>0.1930</td><td>0.1949</td></tr><tr><td>Flan-T5-small</td><td>MINIFOL</td><td>0.3596</td><td>0.2008</td><td>0.26253</td><td>0.1904</td></tr><tr><td>Flan-T5-large*</td><td>CGIF</td><td>0.3251</td><td>0.1828</td><td>0.2308</td><td>0.1811</td></tr><tr><td>Flan-T5-large*</td><td>MINIFOL</td><td>0.2955</td><td>0.1895</td><td>0.2101</td><td>0.1771</td></tr></tbody></table>

<table><caption>**Table 6**
Supervised Fine-Tuning Results on A100 GPU: best results in bold and second best underlined.</caption><thead><tr><th>Model</th><th>Grammar</th><th>Accuracy</th><th>Precision</th><th>Recall</th><th>F1</th></tr></thead><tbody><tr><td>Flan-T5-base</td><td>CLIF</td><td>0.5073</td><td>0.5001</td><td>0.5015</td><td>0.4986</td></tr><tr><td>Flan-T5-base</td><td>NL</td><td>0.5418</td><td>0.5425</td><td>0.5397</td><td>0.5387</td></tr><tr><td>Flan-T5-base</td><td>TFL</td><td>0.4827</td><td>0.6240</td><td>0.4646</td><td>0.4187</td></tr><tr><td>Flan-T5-large</td><td><strong>NL</strong></td><td><strong>0.6600</strong></td><td><strong>0.6622</strong></td><td><strong>0.6572</strong></td><td><strong>0.6585</strong></td></tr><tr><td>Flan-T5-large</td><td><strong>CLIF</strong></td><td><u>0.6157</u></td><td><u>0.6148</u></td><td><u>0.6156</u></td><td><u>0.6149</u></td></tr><tr><td>Flan-T5-base</td><td>FOL</td><td>0.4876</td><td>0.4716</td><td>0.4727</td><td>0.4444</td></tr><tr><td>Flan-T5-base</td><td>TFL+</td><td>0.4926</td><td>0.5690</td><td>0.4775</td><td>0.4504</td></tr><tr><td>Flan-T5-large</td><td>TFL+</td><td>0.5418</td><td>0.5594</td><td>0.5350</td><td>0.5347</td></tr></tbody></table>

quantitative analysis may be needed to explore these results further and may be included in future work along with the analysis of the other syllogistic types which has not been included due to size restriction. The results enable us to answer our SRQs. For SRQ1, the best method shown empirically remains SFT as it provides the highest and most stable results among different representations. For SRQ2, our results show that controlled formal languages generally scale well with models while remaining consistent with performances. For SRQ3, our experiments show that tailoring the vocabulary size of the model tokenizer to that of the language results in erratic performances that do not scale well. Future work will include experiments to evaluate the effect of tokenizer post-retraining on the grammar token distribution for compact languages (e.g. TFL+) to understand the cases where re-training would help or fail.

## 5. Conclusions and Future Work

In this paper, we presented the early stage of a PhD research tackling the formal representation of data in SLMs to possibly bootstrap the construction of ontologies with external knowledge. We focused here on the formalization of logical data with the goal of finding a compact representation that keeps SLMs performing competitively on reasoning tasks. Our experiments enable us to answer our RQ: logical data for first-order reasoning in NL can be challenged by more compact and formal representations. From

<footnote>* The model is fine-tuned with PEFT-LoRA</footnote>
<table><caption>**Table 7**

Zero-Shot Results on L4 GPU. The best results are in bold and the second best are underlined.</caption><thead><tr><th>Model</th><th>Grammar</th><th>Accuracy</th><th>Precision</th><th>Recall</th><th>F1</th></tr></thead><tbody><tr><td>Gemma-2-2b-it</td><td><u>NL</u></td><td>0.4827</td><td>0.4704</td><td>0.4901</td><td><strong>0.4106</strong></td></tr><tr><td>Phi-3.5-mini-instruct</td><td><u>NL</u></td><td>0.4380</td><td>0.5300</td><td>0.4200</td><td><u>0.4000</u></td></tr><tr><td>Flan-T5-large</td><td><strong>NL</strong></td><td><strong>0.4975</strong></td><td>0.3313</td><td><strong>0.5026</strong></td><td>0.3991</td></tr><tr><td>Gemma-2-2b-it</td><td>CLIF</td><td>0.4433</td><td>0.5028</td><td>0.4346</td><td>0.3672</td></tr><tr><td>Phi-3.5-mini-instruct</td><td>NL + FOL</td><td>0.4090</td><td>0.5100</td><td>0.3900</td><td>0.3600</td></tr><tr><td>Gemma-2-2b-it</td><td>FOL</td><td>0.4334</td><td>0.4989</td><td>0.4233</td><td>0.3574</td></tr><tr><td>Gemma-2-2b-it</td><td>MINIFOL</td><td>0.3694</td><td>0.4081</td><td>0.3584</td><td>0.2937</td></tr><tr><td>Flan-T5-large</td><td>CLIF</td><td>0.3891</td><td>0.2899</td><td>0.3769</td><td>0.2828</td></tr><tr><td>Flan-T5-small</td><td>NL</td><td>0.3990</td><td>0.2959</td><td>0.3824</td><td>0.2669</td></tr><tr><td>GPT-2</td><td>FOL</td><td>0.3103</td><td>0.2993</td><td>0.3250</td><td>0.2660</td></tr><tr><td>Gemma-2-2b-it</td><td>CGIF</td><td>0.3399</td><td>0.3276</td><td>0.3260</td><td>0.2657</td></tr><tr><td>Phi-3.5-mini-instruct</td><td>FOL</td><td>0.3550</td><td>0.2400</td><td>0.3400</td><td>0.2500</td></tr><tr><td>GPT-2</td><td>NL</td><td>0.2758</td><td>0.2817</td><td>0.2881</td><td>0.2499</td></tr><tr><td>Gemma-2-2b-it</td><td><strong>TFL+</strong></td><td>0.3399</td><td><strong>0.5634</strong></td><td>0.3310</td><td>0.2498</td></tr><tr><td>GPT-2</td><td><u>NL + FOL</u></td><td>0.3152</td><td><u>0.5444</u></td><td>0.3335</td><td>0.2494</td></tr><tr><td>Flan-T5-large</td><td>CGIF</td><td>0.3399</td><td>0.2001</td><td>0.3261</td><td>0.2287</td></tr><tr><td>Flan-T5-large</td><td>NL + FOL</td><td>0.3596</td><td>0.1830</td><td>0.2635</td><td>0.2034</td></tr><tr><td>Gemma-2-2b-it</td><td>TFL</td><td>0.3399</td><td>0.2225</td><td>0.2419</td><td>0.1863</td></tr><tr><td>Flan-T5-small</td><td>FOL</td><td>0.3546</td><td>0.2294</td><td>0.3340</td><td>0.1842</td></tr><tr><td>Flan-T5-small</td><td>NL + FOL</td><td>0.3546</td><td>0.1182</td><td>0.3333</td><td>0.1745</td></tr><tr><td>Flan-T5-large</td><td>TFL</td><td>0.3546</td><td>0.1182</td><td>0.3333</td><td>0.1745</td></tr><tr><td>Flan-T5-large</td><td>MINIFOL</td><td>0.3497</td><td>0.0777</td><td>0.1012</td><td>0.0787</td></tr><tr><td>Flan-T5-large</td><td>FOL</td><td>0.3300</td><td>0.0658</td><td>0.1332</td><td>0.0779</td></tr><tr><td>Flan-T5-small</td><td>MINIFOL</td><td>0</td><td>0</td><td>0</td><td>0</td></tr><tr><td>Flan-T5-small</td><td>TFL</td><td>0</td><td>0</td><td>0</td><td>0</td></tr></tbody></table>

<table><caption>**Table 8**

Zero-Shot with and without BNF Grammar Prompting Results on L4 GPU. The best results are in bold and the second best are underlined.</caption><thead><tr><th>Model</th><th>Grammar</th><th>Accuracy</th><th>Precision</th><th>Recall</th><th>F1</th><th>Grammar Prompting</th></tr></thead><tbody><tr><td>Gemma-2-2b-it</td><td><strong>CLIF</strong></td><td><strong>0.4532</strong></td><td>0.4633</td><td><strong>0.4368</strong></td><td><strong>0.3924</strong></td><td>Yes</td></tr><tr><td>Gemma-2-2b-it</td><td><u>CLIF</u></td><td><u>0.4433</u></td><td><u>0.5028</u></td><td><u>0.4346</u></td><td><u>0.3672</u></td><td>No</td></tr><tr><td>Gemma-2-2b-it</td><td>FOL</td><td>0.4187</td><td>0.4021</td><td>0.4018</td><td>0.3514</td><td>Yes</td></tr><tr><td>Gemma-2-2b-it</td><td>FOL</td><td>0.4334</td><td>0.4989</td><td>0.4233</td><td>0.3574</td><td>No</td></tr><tr><td>Gemma-2-2b-it</td><td>TFL+</td><td>0.3596</td><td>0.3618</td><td>0.3445</td><td>0.2782</td><td>Yes</td></tr><tr><td>Gemma-2-2b-it</td><td><strong>TFL+</strong></td><td>0.3399</td><td><strong>0.5634</strong></td><td>0.3310</td><td>0.2498</td><td>No</td></tr><tr><td>Gemma-2-2b-it</td><td>TFL</td><td>0.3645</td><td>0.4689</td><td>0.3458</td><td>0.2422</td><td>Yes</td></tr><tr><td>Gemma-2-2b-it</td><td>TFL</td><td>0.3399</td><td>0.2225</td><td>0.2419</td><td>0.1863</td><td>No</td></tr></tbody></table>

our empirical results, CLIF emerged as a strong candidate due to the compactness of its vocabulary, its easy transformation to and from FOL and its competitive results especially in a SFT setting. The results were achieved on small, frugal language models that are all under 3 billion parameters, making them accessible for reasoning tasks over logical data. While our experiments do not point to an absolute best representation that overtakes NL in the formalization of logical data, they show that expressing inputs in compact formal languages comes very close for SLMs on reasoning tasks. This takeaway allows us to focus on the next goal of this PhD research which is to leverage SLMs and formal representations of ontological data to bootstrap ontology construction.
<table><caption>**Table 9**
8-Shot Results on L4 GPU. The best results are in bold and the second best are underlined.</caption><thead><tr><th>Model</th><th>Grammar</th><th>Accuracy</th><th>Precision</th><th>Recall</th><th>F1</th></tr></thead><tbody><tr><td>Flan-T5-large</td><td>NL</td><td><strong>0.4926</strong></td><td>0.3313</td><td><strong>0.5002</strong></td><td><strong>0.3959</strong></td></tr><tr><td>Flan-T5-large</td><td>CLIF</td><td><u>0.4334</u></td><td>0.3302</td><td><u>0.4223</u></td><td><u>0.3269</u></td></tr><tr><td>Flan-T5-large</td><td>NL + FOL</td><td>0.3940</td><td>0.2773</td><td>0.3853</td><td>0.2989</td></tr><tr><td>Flan-T5-large</td><td>MINIFOL</td><td>0.3891</td><td>0.2485</td><td>0.3821</td><td>0.2963</td></tr><tr><td>Phi-3.5-mini-instruct</td><td>NL</td><td>0.2710</td><td><strong>0.5200</strong></td><td>0.2700</td><td>0.2700</td></tr><tr><td>Phi-3.5-mini-instruct</td><td>FOL</td><td>0.2320</td><td><u>0.3700</u></td><td>0.2300</td><td>0.2600</td></tr><tr><td>Flan-T5-small</td><td>NL</td><td>0.3940</td><td>0.3674</td><td>0.3763</td><td>0.2548</td></tr><tr><td>Flan-T5-large</td><td>FOL</td><td>0.3546</td><td>0.2107</td><td>0.3355</td><td>0.2016</td></tr><tr><td>Flan-T5-large</td><td>TFL</td><td>0.3251</td><td>0.2045</td><td>0.3085</td><td>0.1934</td></tr><tr><td>Flan-T5-small</td><td>FOL</td><td>0.3546</td><td>0.1182</td><td>0.3333</td><td>0.1745</td></tr><tr><td>Flan-T5-small</td><td>NL + FOL</td><td>0.3546</td><td>0.1182</td><td>0.3333</td><td>0.1745</td></tr><tr><td>GPT-2</td><td>FOL</td><td>0.3546</td><td>0.1182</td><td>0.3333</td><td>0.1745</td></tr><tr><td>Gemma-2-2b-it</td><td>NL</td><td>0.3546</td><td>0.1182</td><td>0.3333</td><td>0.1745</td></tr><tr><td>Gemma-2-2b-it</td><td>TFL</td><td>0.3546</td><td>0.1182</td><td>0.3333</td><td>0.1745</td></tr><tr><td>Gemma-2-2b-it</td><td>TFL+</td><td>0.3546</td><td>0.1182</td><td>0.3333</td><td>0.1745</td></tr><tr><td>Gemma-2-2b-it</td><td>FOL</td><td>0.3546</td><td>0.1182</td><td>0.3333</td><td>0.1745</td></tr><tr><td>Gemma-2-2b-it</td><td>CLIF</td><td>0.3546</td><td>0.1182</td><td>0.3333</td><td>0.1745</td></tr><tr><td>Gemma-2-2b-it</td><td>CGIF</td><td>0.3546</td><td>0.1182</td><td>0.3333</td><td>0.1745</td></tr><tr><td>GPT-2</td><td>NL</td><td>0.3399</td><td>0.1133</td><td>0.3333</td><td>0.1691</td></tr><tr><td>GPT-2</td><td>NL + FOL</td><td>0.3054</td><td>0.1018</td><td>0.3333</td><td>0.1559</td></tr><tr><td>Flan-T5-large</td><td>CGIF</td><td>0.3448</td><td>0.1467</td><td>0.2447</td><td>0.1488</td></tr><tr><td>Phi-3.5-mini-instruct</td><td>NL + FOL</td><td>0.1380</td><td>0.1200</td><td>0.1400</td><td>0.1300</td></tr><tr><td>Flan-T5-small</td><td>MINIFOL</td><td>0.1231</td><td>0.0054</td><td>0.0041</td><td>0.0047</td></tr><tr><td>Flan-T5-small</td><td>TFL</td><td>0</td><td>0</td><td>0</td><td>0</td></tr></tbody></table>

<table><caption>**Table 10**
8-Shot with and without BNF Grammar Prompting Results on L4 GPU.</caption><thead><tr><th>Model</th><th>Grammar</th><th>Accuracy</th><th>Precision</th><th>Recall</th><th>F1</th><th>Grammar Prompting</th></tr></thead><tbody><tr><td>Gemma-2-2b-it</td><td>TFL</td><td>0.3546</td><td>0.1182</td><td>0.3333</td><td>0.1745</td><td>Yes</td></tr><tr><td>Gemma-2-2b-it</td><td>TFL</td><td>0.3546</td><td>0.1182</td><td>0.3333</td><td>0.1745</td><td>No</td></tr><tr><td>Gemma-2-2b-it</td><td>TFL+</td><td>0.3546</td><td>0.1182</td><td>0.3333</td><td>0.1745</td><td>Yes</td></tr><tr><td>Gemma-2-2b-it</td><td>TFL+</td><td>0.3546</td><td>0.1182</td><td>0.3333</td><td>0.1745</td><td>No</td></tr><tr><td>Gemma-2-2b-it</td><td>FOL</td><td>0.3546</td><td>0.1182</td><td>0.3333</td><td>0.1745</td><td>Yes</td></tr><tr><td>Gemma-2-2b-it</td><td>FOL</td><td>0.3546</td><td>0.1182</td><td>0.3333</td><td>0.1745</td><td>No</td></tr><tr><td>Gemma-2-2b-it</td><td>CLIF</td><td>0.3546</td><td>0.1182</td><td>0.3333</td><td>0.1745</td><td>Yes</td></tr><tr><td>Gemma-2-2b-it</td><td>CLIF</td><td>0.3546</td><td>0.1182</td><td>0.3333</td><td>0.1745</td><td>No</td></tr></tbody></table>

## 5.1. Next Activities in this PhD Research

This PhD research will explore mixed-input representations (e.g. NL + CLIF) to assess whether combining expressiveness with the structure of a formal grammar improves reasoning. We will also explore injecting knowledge from a high-level ontology (e.g. DOLCE [39]) in a formal language to efficiently reuse relevant knowledge learned by the model in an ontology extension pipeline.

## 5.2. Related Future Work Directions

We identify several future research directions including the extension of the SEF-CLGC evaluation to datasets such as ProofWriter<fnref n="4" />, RuleTaker<fnref n="5" />, the Logical Entailment Dataset<fnref n="6" /> and SynLogic<fnref n="7" />. Another

<footnote n="4">⁴https://www.kaggle.com/datasets/mathurinache/proofwriter</footnote>

<footnote n="5">⁵https://github.com/allenai/ruletaker</footnote>

<footnote n="6">⁶https://github.com/google-deepmind/logical-entailment-dataset</footnote>

<footnote n="7">⁷https://github.com/MiniMax-AI/SynLogic</footnote>
<caption>**Table 11**</caption>

Supervised Fine-Tuning with and without Grammar Context-Passing Results. The best results are in bold and the second best are underlined.

<table><thead><tr><th>Model</th><th>Grammar</th><th>Accuracy</th><th>Precision</th><th>Recall</th><th>F1</th><th>ICGP<sup>†</sup></th><th>GPU</th></tr></thead><tbody><tr><td>Flan-T5-large*</td><td>TFL</td><td>0.3546</td><td>0.1182</td><td>0.3333</td><td>0.1745</td><td>Yes</td><td>L4</td></tr><tr><td>Flan-T5-large*</td><td>TFL</td><td>0.4334</td><td>0.2858</td><td>0.3112</td><td>0.2962</td><td>No</td><td>L4</td></tr><tr><td>Flan-T5-small</td><td>CLIF</td><td>0.3694</td><td>0.3131</td><td>0.3486</td><td>0.2299</td><td>Yes</td><td>L4</td></tr><tr><td>Flan-T5-small</td><td><u>CLIF</u></td><td><u>0.4384</u></td><td>0.4425</td><td><u>0.4274</u></td><td><u>0.4109</u></td><td>No</td><td>L4</td></tr><tr><td>Flan-T5-base</td><td><b>CLIF</b></td><td>0.4334</td><td><b>0.6252</b></td><td>0.4129</td><td>0.3317</td><td>Yes</td><td>A100</td></tr><tr><td>Flan-T5-base</td><td><b>CLIF</b></td><td><b>0.5073</b></td><td><u>0.5001</u></td><td><b>0.5015</b></td><td><b>0.4986</b></td><td>No</td><td>A100</td></tr></tbody></table>

<caption>**Table 12**</caption>

Supervised Fine-Tuning with Tokenizer Re-Training Results. The best results are in bold and the second best are underlined.

<table><caption>**Table 13**</caption><thead><tr><th>Model</th><th>Grammar</th><th>Accuracy</th><th>Precision</th><th>Recall</th><th>F1</th><th>GPU</th><th>Re-Train Tokenizer</th><th>Vocabulary Size</th></tr></thead><tbody><tr><td>Flan-T5-small</td><td>TFL</td><td>0.3201</td><td>0.2983</td><td>0.3109</td><td>0.2790</td><td>T4/L4</td><td>Yes</td><td>191</td></tr><tr><td>Flan-T5-small</td><td>TFL</td><td>0.3596</td><td>0.3424</td><td>0.3476</td><td>0.3070</td><td>T4/L4</td><td>No</td><td>32128</td></tr><tr><td>Flan-T5-small</td><td>CLIF</td><td>0.3497</td><td>0.3600</td><td>0.3466</td><td>0.3382</td><td>T4/L4</td><td>Yes</td><td>32128</td></tr><tr><td>Flan-T5-small</td><td>CLIF</td><td>0.4384</td><td>0.4425</td><td>0.4274</td><td>0.4109</td><td>T4/L4</td><td>No</td><td>32128</td></tr><tr><td>Flan-T5-small</td><td>TFL+</td><td>0.4532</td><td>0.4286</td><td>0.4395</td><td>0.4113</td><td>T4/L4</td><td>Yes</td><td>180</td></tr><tr><td>Flan-T5-small</td><td>TFL+</td><td>0.3596</td><td>0.3243</td><td>0.3440</td><td>0.2918</td><td>T4/L4</td><td>No</td><td>32128</td></tr><tr><td>Flan-T5-base</td><td>TFL+</td><td>0.4334</td><td>0.4083</td><td>0.4168</td><td>0.3713</td><td>A100</td><td>Yes</td><td>180</td></tr><tr><td>Flan-T5-base</td><td><u>TFL+</u></td><td><u>0.4926</u></td><td><b>0.5690</b></td><td><u>0.4775</u></td><td><u>0.4504</u></td><td>A100</td><td>No</td><td>32128</td></tr><tr><td>Flan-T5-large</td><td>TFL+</td><td>0.4039</td><td>0.4786</td><td>0.3854</td><td>0.3167</td><td>A100</td><td>Yes</td><td>180</td></tr><tr><td>Flan-T5-large</td><td><u>TFL+</u></td><td><b>0.5418</b></td><td><u>0.5594</u></td><td><b>0.5350</b></td><td><b>0.5347</b></td><td>A100</td><td>No</td><td>32128</td></tr></tbody></table>

Syllogism Evaluation Framework for Supervised Fine-Tuning Flan-T5-large on NL, CLIF and TFL+.

<table><thead><tr><th rowspan="2">Grammar</th><th colspan="2">Hypothetical</th><th colspan="2">Disjunctive</th><th colspan="2">Complex</th><th colspan="2">Categorical</th></tr><tr><th>Hit</th><th>Miss</th><th>Hit</th><th>Miss</th><th>Hit</th><th>Miss</th><th>Hit</th><th>Miss</th></tr></thead><tbody><tr><td>NL</td><td>59</td><td>25</td><td>60</td><td>39</td><td>14</td><td>2</td><td>2</td><td>2</td></tr><tr><td>CLIF</td><td>48</td><td>36</td><td>57</td><td>42</td><td>14</td><td>2</td><td>2</td><td>2</td></tr><tr><td>TFL+</td><td>44</td><td>40</td><td>53</td><td>46</td><td>15</td><td>1</td><td>2</td><td>2</td></tr></tbody></table>

direction would be using LMs to extend the vocabulary of an ontology to handle new scenarios given Competency Questions (CQs) and the original ontology in practical use cases like HMAS<fnref n="8" /> using categorical syllogisms to explain the hierarchical connections between generated classes and properties.

## Acknowledgments

This work is supported by 3IA Côte d'Azur (ANR-19-P3IA-0002), UCAJEDI (ANR-15-IDEX-01), Université Côte d'Azur's Center for High-Performance Computing and Data ScienceTech Institute.

## Declaration on Generative AI

The author(s) have not employed any Generative AI tools.

<footnote>* The model is fine-tuned with PEFT-LoRA</footnote>

<footnote>† ICGP = In-Context Grammar Passing</footnote>

<footnote n="8">⁸https://github.com/HyperAgents/hmas</footnote>
<table><caption>**Table 14**
Flan-T5-large model SFT performance on NL, CLIF and TFL+ for Categorical syllogisms.</caption><thead><tr><th>Premises</th><th>Conclusion</th><th>True Label</th><th>NL Label</th><th>CLIF Label</th><th>TFL+ Label</th></tr></thead><tbody><tr><td>Some affection is love. Some love is positive.</td><td>Some affection is positive.</td><td>Uncertain</td><td>True</td><td>True</td><td>True</td></tr><tr><td>Heinrich Schmidt was a German politician. Heinrich Schmidt was also a member of the Prussian State Parliament and the Nazi Reichstag.</td><td>Heinrich Schmidt was German or Russian or both.</td><td>True</td><td>Uncertain</td><td>Uncertain</td><td>False</td></tr><tr><td>Heinrich Schmidt was a German politician. Heinrich Schmidt was also a member of the Prussian State Parliament and the Nazi Reichstag.</td><td>Some German politician was part of both the Prussian State Parliament and the Nazi Reichstag.</td><td>True</td><td>True</td><td>True</td><td>True</td></tr><tr><td>Heinrich Schmidt was a German politician. Heinrich Schmidt was also a member of the Prussian State Parliament and the Nazi Reichstag.</td><td>No politicians are part of the Nazi Reichstag.</td><td>False</td><td>False</td><td>False</td><td>False</td></tr></tbody></table>

## References

[1] Z. Guo, R. Jin, C. Liu, Y. Huang, D. Shi, L. Yu, Y. Liu, J. Li, B. Xiong, D. Xiong, et al., Evaluating large language models: A comprehensive survey, Preprint arXiv:2310.19736 (2023).

[2] S. Minaee, T. Mikolov, N. Nikzad, M. Chenaghlu, R. Socher, X. Amatriain, J. Gao, Large language models: A survey, arxiv 2024, Preprint arXiv:2402.06196 (2024).

[3] H. Liu, Z. Fu, M. Ding, R. Ning, C. Zhang, X. Liu, Y. Zhang, Logical reasoning in large language models: A survey, Preprint arXiv:2502.09100 (2025).

[4] W. Wang, Y. Yang, F. Wu, Towards data-and knowledge-driven ai: a survey on neuro-symbolic computing, IEEE Transactions on Pattern Analysis and Machine Intelligence (2024).

[5] B. Jin, G. Liu, C. Han, M. Jiang, H. Ji, J. Han, Large language models on graphs: A comprehensive survey, IEEE Transactions on Knowledge and Data Engineering (2024).

[6] A. S. Lippolis, M. J. Saeedizade, R. Keskisärkkä, S. Zuppiroli, M. Ceriani, A. Gangemi, E. Blomqvist, A. G. Nuzzolese, Ontology generation using large language models, in: European Semantic Web Conference, Springer, 2025, pp. 321–341.

[7] Y. Zhao, B. Zhang, X. Hu, S. Ouyang, J. Kim, N. Jain, J. de Berardinis, A. Meroño-Peñuela, E. Simperl, Improving ontology requirements engineering with ontochat and participatory prompting, in: Proceedings of the AAAI Symposium Series, volume 4, 2024, pp. 253–257.

[8] Y. Zhao, N. Vetter, K. Aryan, Using large language models for ontoclean-based ontology refinement, Preprint arXiv:2403.15864 (2024).

[9] Y. Zhao, Leveraging large language models for ontology requirements engineering, in: Extended Semantic Web Conference ESWC, 2025, pp. –.

[10] N. Fathallah, A. Das, S. D. Giorgis, A. Poltronieri, P. Haase, L. Kovriguina, Neon-gpt: a large language model-powered pipeline for ontology learning, in: European Semantic Web Conference, Springer, 2024, pp. 36–50.

[11] Z. Hou, Neural-symbolic reasoning: Towards the integration of logical reasoning with large language models, Authorea Preprints (2025).

[12] V. Schlegel, K. V. Pavlov, I. Pratt-Hartmann, Can transformers reason in fragments of natural language?, Preprint arXiv:2211.05417 (2022).

[13] J. Huang, K. C.-C. Chang, Towards reasoning in large language models: A survey, Preprint arXiv:2212.10403 (2022).

[14] G. Srivastava, S. Cao, X. Wang, Towards reasoning ability of small language models, Preprint arXiv:2502.11569 (2025).

[15] J. Bi, Y. Wu, W. Xing, Z. Wei, Enhancing the reasoning capabilities of small language models via solution guidance fine-tuning, Preprint arXiv:2412.09906 (2024).
[16] Y. Kim, E. Yi, M. Kim, S.-Y. Yun, T. Kim, Guiding reasoning in small language models with llm assistance, Preprint arXiv:2504.09923 (2025).

[17] M. Besta, J. Barth, E. Schreiber, A. Kubicek, A. Catarino, R. Gerstenberger, P. Nyczyk, P. Iff, Y. Li, S. Houliston, et al., Reasoning language models: A blueprint, Preprint arXiv:2501.11223 (2025).

[18] K. Kumar, T. Ashraf, O. Thawakar, R. M. Anwer, H. Cholakkal, M. Shah, M.-H. Yang, P. H. Torr, F. S. Khan, S. Khan, Llm post-training: A deep dive into reasoning large language models, Preprint arXiv:2502.21321 (2025).

[19] H. R. Faghihi, Q. Guo, A. Uszok, A. Nafar, E. Raisi, P. Kordjamshidi, Domiknows: A library for integration of symbolic domain knowledge in deep learning, Preprint arXiv:2108.12370 (2021).

[20] P. Kesseli, P. O'Hearn, R. S. Cabral, Logic. py: Bridging the gap between llms and constraint solvers, Preprint arXiv:2502.15776 (2025).

[21] S. Hao, S. Sukhbaatar, D. Su, X. Li, Z. Hu, J. Weston, Y. Tian, Training large language models to reason in a continuous latent space, 2024, URL https://arxiv.org/abs/2412.06769 (2022).

[22] F. Cheng, H. Li, F. Liu, R. van Rooij, K. Zhang, Z. Lin, Empowering llms with logical reasoning: A comprehensive survey, Preprint arXiv:2502.15652 (2025).

[23] S. Han, H. Schoelkopf, Y. Zhao, Z. Qi, M. Riddell, W. Zhou, J. Coady, D. Peng, Y. Qiao, L. Benson, et al., Folio: Natural language reasoning with first-order logic, in: Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 2024, pp. 22017–22031.

[24] Y.-F. Zhang, H. Zhang, L. E. Li, E. Xing, Evaluating step-by-step reasoning through symbolic verification, Preprint arXiv:2212.08686 (2022).

[25] J. Xu, H. Fei, L. Pan, Q. Liu, M.-L. Lee, W. Hsu, Faithful logical reasoning via symbolic chain-of-thought, Preprint arXiv:2405.18357 (2024).

[26] S. Yang, E. Gribovskaya, N. Kassner, M. Geva, S. Riedel, Do large language models latently perform multi-hop reasoning?, Preprint arXiv:2402.16837 (2024).

[27] S. Yang, N. Kassner, E. Gribovskaya, S. Riedel, M. Geva, Do large language models perform latent multi-hop reasoning without exploiting shortcuts?, Preprint arXiv:2411.16679 (2024).

[28] B. Ghosh, S. Hasan, N. A. Arafat, A. Khan, Logical consistency of large language models in fact-checking, Preprint arXiv:2412.16100 (2024).

[29] Y. Peng, Z. Long, F. Dong, C. Li, S. Wu, K. Chen, Playing language game with llms leads to jailbreaking, Preprint arXiv:2411.12762 (2024).

[30] Y. Wu, M. Han, Y. Zhu, L. Li, X. Zhang, R. Lai, X. Li, Y. Ren, Z. Dou, Z. Cao, Hence, socrates is mortal: A benchmark for natural language syllogistic reasoning, in: Findings of the Association for Computational Linguistics: ACL 2023, 2023, pp. 2347–2367.

[31] J. F. Sowa, Conceptual graphs, Foundations of artificial intelligence 3 (2008) 213–237.

[32] J. F. Sowa, Conceptual graphs as a universal knowledge representation, Computers & Mathematics with Applications 23 (1992) 75–93.

[33] J. Sowa, Introduction to common logic, 2011.

[34] F. Sommers, G. Englebretsen, An invitation to formal reasoning: the logic of terms, Routledge, 2017.

[35] J. M. Castro-Manzano, L. I. Lozano-Cobos, P. O. Reyes-Cardenas, Programming with term logic, BRAIN. Broad Research in Artificial Intelligence and Neuroscience 9 (2018) 22–36.

[36] J. M. C. Manzano, An intermediate term functor logic, Argumentos - Revista de Filosofia 11 (2025) 17–31.

[37] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, W. Chen, et al., Lora: Low-rank adaptation of large language models., ICLR 1 (2022) 3.

[38] B. Wang, Z. Wang, X. Wang, Y. Cao, R. A Saurous, Y. Kim, Grammar prompting for domain-specific language generation with large language models, Advances in Neural Information Processing Systems 36 (2023) 65030–65055.

[39] S. Borgo, R. Ferrario, A. Gangemi, N. Guarino, C. Masolo, D. Porello, E. M. Sanfilippo, L. Vieu, Dolce: A descriptive ontology for linguistic and cognitive engineering, Applied ontology 17 (2022) 45–69.