<figure><img src="image_1.png" /></figure>

HAL
open science

# Text2KGBench-LettrIA: A refined benchmark for Text2Graph systems

Julien Plu, Oscar Moreno Escobar, Edouard Trouillez, Axelle Gapin, Pasquale

Lisena, Thibault Ehrhart, Raphaël Troncy

<figure><img src="image_2.png" /></figure>

## To cite this version:

Julien Plu, Oscar Moreno Escobar, Edouard Trouillez, Axelle Gapin, Pasquale Lisena, et al. Text2KGBench-LettrIA: A refined benchmark for Text2Graph systems. KBC-LM and LM-KBC Challenge (Knowledge Base Construction from Pre-trained Language Models) at ISWC 2025, 24th International Semantic Web Conference, CEUR, Nov 2025, Nara, Japan. hal-05301914

HAL Id: hal-05301914

https://hal.science/hal-05301914v1

Submitted on 7 Oct 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_3.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Text2KGBench-LettrIA: A Refined Benchmark for Text2Graph Systems

Julien Plu<fnref n="1" />, Oscar Moreno Escobar<fnref n="1" />, Edouard Trouillez<fnref n="1" />, Axelle Gapin<fnref n="1" />,†, Pasquale Lisena<fnref n="2" />, Thibault Ehrhart<fnref n="2" /> and Raphaël Troncy<fnref n="2" />

## Abstract

Recent advances in Large Language Models (LLMs) have catalyzed significant research into automated knowledge graph (KG) construction from text, a fundamental challenge at the intersection of natural language processing and semantic web technologies. However, the reliability of evaluating model performance is hindered by limitations in existing benchmarks like Text2KGBench, which exhibit shortcomings in data quality, ontological consistency, and structural design. To address these issues, this paper introduces Text2KGBench-LettrIA, a substantially revised and curated benchmark derived from the DBpedia-WebNLG portion of Text2KGBench. Our primary contributions include: (1) the systematic refinement of 19 domain ontologies to enforce hierarchical structure and formal typing; (2) a complete re-annotation of 4,860 sentences, yielding over 14,000 high-fidelity triples under a strict set of annotation guidelines; and (3) the introduction of an enriched data format with enhanced metadata to ensure reproducibility and support multifaceted evaluation. We demonstrate the utility of our benchmark by evaluating a suite of both proprietary and open-weights LLMs in zero-shot and fine-tuned settings, respectively. Our results reveal a key finding: smaller, fine-tuned open-weights models can achieve superior F1 accuracy compared to their larger, proprietary counterparts, underscoring the critical role of high-quality, schema-aligned training data.

## 1. Introduction

The recent proliferation of Large Language Models (LLMs) and foundation models has catalyzed significant advancements in Natural Language Processing (NLP). A key area of impact is the automated construction and completion of Knowledge Graphs (KGs), where the synergy between LLMs and structured knowledge is pivotal. This relationship underpins emerging applications such as the generation of explainable AI (XAI) outputs and the development of robust neuro-symbolic fact-checking systems. A significant contribution in this domain is Text2KGBench [1], a benchmark designed to evaluate the capacity of language models to generate KGs from text under ontological guidance. The framework assesses a model's ability to extract relational triples that both conform to a predefined ontology and remain grounded in the source text. Text2KGBench is composed of two datasets: Wikidata-TekGen, derived from the TekGen corpus [2], containing 13,474 sentences across 10 ontologies; and DBpedia-WebNLG, based on the WebNLG corpus [3], with 4860 sentences across 19 ontologies. Both split in training and test set. Despite its foundational role, a detailed analysis of Text2KGBench reveals several critical limitations that hinder reliable model evaluation and impede progress. Our investigation, which focuses on the DBpedia-WebNLG component, identifies the following principal flaws:

* **Ontological:** The ontologies are semantically imprecise. They suffered from a flat, non-hierarchical design, contained ambiguous and out-of-domain concepts, and lacked the formal rigor needed for robust knowledge representation, making it difficult to use for precise model evaluation and knowledge extraction.

<figure><img src="image_4.png" /></figure>

<figure><img src="image_5.png" /></figure>

<figure><img src="image_6.png" /></figure>

<footnote n="1">¹LettrIA, Paris, France</footnote>

<footnote n="2">²EURECOM, Sophia Antipolis, France</footnote>

<footnote>Joint proceedings of KBC-LM and LM-KBC @ ISWC 2025</footnote>

<footnote>† Author contributed as consulting.</footnote>

<footnote>julien@lettria.com (J. Plu); oscar@lettria.com (O. M. Escobar); edouard@lettria.com (E. Trouillez);</footnote>

<footnote>pasquale.lisena@eurecom.fr (P. Lisena); thibault.ehrhart@eurecom.fr (T. Ehrhart); raphael.troncy@eurecom.fr (R. Troncy)</footnote>

<footnote>0000-0002-7876-3441 (J. Plu); 0000-0003-3094-5585 (P. Lisena); 0000-0003-1377-8279 (T. Ehrhart); 0000-0003-0457-1436</footnote>

<footnote>(R. Troncy)</footnote>

<footnote>© 2025 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).</footnote>
* **Annotation and Data Quality:** The data annotations in the original benchmark were inconsistent and unreliable. This was caused by a lack of standardization for entity names and literal values, a failure to strictly limit annotations to textual evidence, and the presence of grammatical errors in the source sentences.
* **Structural and Technical:** From a technical perspective, the original dataset was difficult to use and lacked features essential for reproducibility. Its data structure was missing key information and contained formatting errors, while the ontologies themselves was undocumented and used an overly complicated URI scheme.

To address these shortcomings, this work makes the following primary contributions:

* We introduce **Text2KGBench-LettrIA**, a rigorously **corrected and enriched benchmark** for ontology-guided KG construction. This new version rectifies annotation errors, ensures ontological compliance, and improves overall data quality to facilitate more accurate and meaningful model evaluation. The benchmark is available upon request to the authors.
* We conduct an extensive **empirical evaluation of diverse language models**, including proprietary APIs and open-weights models, on Text2KGBench-LettrIA. Our findings reveal that fine-tuned open models can consistently outperform larger, proprietary models in zero- or few-shot settings, demonstrating their effectiveness for structured information extraction.

The remainder of this paper is organized as follows. Section 2 reviews related work on KG construction from text. Section 3 details our methodology for revising the benchmark. Section 4 presents our experimental setup and comparative results. Finally, Section 5 concludes with a summary of our findings and outlines directions for future research.

## 2. Related Work

The task of automatically constructing Knowledge Graphs (KGs) from unstructured text, commonly known as Text-to-Knowledge-Graph (Text2KG), has become a central challenge in natural language processing and semantic web research. This process facilitates the transformation of textual information into structured, machine-readable knowledge representations. It is a composite task that typically integrates sub-problems such as Named Entity Recognition (NER), Relation Extraction (RE), and Entity Linking (EL), which are orchestrated within either pipeline or end-to-end architectures. For a comprehensive formalization of the problem and an extensive literature review, we direct the reader to the systematic survey by Regino et al. [4].

The growing interest in this field is evidenced by sustained community efforts, including the Text2KG workshop series, held annually since 2022 and approaching its fifth edition in 2025 [5], and the yearly Knowledge Base Construction from Pre-trained Language Models (LM-KBC) challenge [6].

These efforts are supported by the development of standardized datasets. One of the earliest and most influential is WebNLG [3], which pairs textual descriptions with RDF-style triples. WebNLG inspired subsequent work like TekGen [2], which expanded the corpus with synthetically generated data. More recently, Text2KGBench [1] established a benchmark to evaluate the generation of ontology-compliant triples grounded in source text. However, as we will detail, Text2KGBench exhibits limitations concerning data quality and ontological rigor, which directly motivates the development of our proposed benchmark. Another significant contribution in this domain is the REBEL dataset [7], which was specifically designed to advance the task of open relation extraction from unstructured text. REBEL introduces a large-scale, fine-grained benchmark that captures a wide range of relations and entities, enabling more comprehensive evaluations of models' ability to extract structured knowledge from natural language.
Methodologies for relation extraction have evolved significantly. Early approaches progressed from rule-based systems to feature-engineered machine learning and subsequently to deep learning architectures. Seminal neural models introduced sequence labeling and multi-task learning frameworks [8]. More advanced architectures like Seq2RDF [9] later framed the task as a sequence-to-sequence problem to translate natural language directly into RDF triples. The advent of transformer-based encoders led to powerful models for joint entity and relation extraction [10]. A critical shortcoming of many of these models, however, is their frequent lack of explicit integration with ontological constraints, limiting their utility for constructing semantically coherent KGs.

To address this gap, the paradigm of schema-aware extraction has emerged, where generated triples must conform to a predefined ontology. Recent studies have explored leveraging external schema constraints during training, for example through few-shot perspective transfer [11] or knowledge-driven synthetic data generation for zero-shot extraction [12]. Others have investigated the use of structured prompts or ontology-guided decoding to improve the alignment of LLM outputs with a target schema. For instance, Ding et al. [13] proposed model collaboration strategies to mitigate hallucinations and enhance recall.

Large Language Models (LLMs) such as GPT-4 and Claude have demonstrated impressive in-context learning capabilities for information extraction. Nonetheless, their application to Text2KG is hampered by a propensity for factual hallucination and inconsistent adherence to structured output formats [14, 5]. While efforts to evaluate and mitigate these issues are ongoing, existing benchmarks often lack the ontological precision required for a fair and rigorous assessment. The benchmark introduced in this paper is specifically designed to fill this void.

# 3. Revision of Text2KGBench

This section details the revision and re-annotation of the Text2KGBench benchmark, undertaken to address critical limitations in its original version and enhance its utility for evaluating modern text-to-graph models. Our efforts focused on two key areas: a comprehensive revision of the underliying ontologies and a complete re-annotation of the corpus based on a new, rigorous set of guidelines. A team of four experts specializing in knowledge representation and natural language processing conducted both activities. The process began with an independent pass by each expert, followed by a reconciliation phase to resolve disagreements by discussing and finding majority consensus on a particular solution. Once all individual annotations were complete, the team convened to review the entire set, discuss any discrepancies, and reach a final consensus.

## 3.1. Ontologies Refinement

The original Text2KGBench ontologies, while extensive, suffered from structural and semantic issues that limited its precision. It was organized into 19 ontologies, one for each domain, but lacked hierarchical depth and formal consistency. We conducted a thorough revision to address these limitations, focusing on improving its coherence, structural integrity, and semantic expressiveness.

**Semantic Coherence and Granularity** A primary objective was to ensure each domain ontology was self-contained and conceptually coherent. We systematically identified and pruned concepts and relations not directly relevant to their specified domains. For example, within the Film ontology, entities such as Club and Station, and relations like spokenIn, were removed as they are better situated in other contexts. This curation ensures that each domain ontology accurately models its core concepts, improving the benchmark's overall focus. The reason behind this decision is because we are internally using ontologies that are focus on a specific client domain or use-case and we do not want extract informations that are not related to this specific client domain or use-case.

To reduce ambiguity and improve clarity, we harmonized property names. For instance, the property campus was renamed to address to more accurately reflect its semantic role, and staff was specified
as academicStaffSize for explicitness. Similarly, the generic location property was refined into
more specific relations such as city or country, depending of the context, thereby increasing the
precision of the knowledge graph.

**Structural and Formal Enhancements** A significant structural enhancement was the introduction of a formal class hierarchy using rdfs:subClassOf relationships. In the original flat structure, University was an isolated class. It is now explicitly defined as a subclass of AcademicInstitution, which itself is a subclass of Organization. This hierarchical structure is not merely a formal improvement; it enables more nuanced evaluation metrics. For instance, we can now measure hierarchical precision, rewarding a model for predicting a correct superclass (e.g. AcademicInstitution even if the specific subclass University) is missed.

Further, properties were rigorously typed as either **ObjectProperty** (linking two entities) and **DatatypeProperty** (linking an entity to a literal value), with explicit domains and ranges defined for each. Datatype ranges were specified using standard XML Schema types (e.g. xsd:string, xsd date, or xsd:integer), enforcing data consistency and aiding downstream processing. To improve usability, we added rdfs:comment annotations for all properties and classes and simplified the URIs by removing the intermediate relations and /concepts path segments. The rdfs:comment annotations have been generated by the authors altogether with our own words to define each annotations.

Finally, to support reproducibility and tracking, the new ontology includes metadata for contributors and is explicitly versioned as version 2.0 using owl:versionIRI. A comprehensive comparison of these changes is presented in Table 4, in the Appendix.

In the appendix, Table 2 presents an overview of the main statistics for each ontology in Text2KGBench-LettrIA and Text2KGBench. The Text2KGBench-LettrIA dataset is significantly lighter, with approximately 21.80% fewer classes and approximately 37.81% fewer properties. Additionally, datatype properties are exclusively present in Text2KGBench-LettrIA.

**3.2. Re-annotations Guidelines**

A robust benchmark requires annotation guidelines that are consistent, unambiguous, and computationally tractable. We established a comprehensive rulebook for the re-annotation process to ensure high-quality, reproducible data.

**Normalization of Literals** To ensure uniformity, we normalized literal values. Dates are standardized to the ISO 8601 format (yyyy-mm-dd). Ambiguous formats like xx/xx/xxxx are interpreted as mm/dd/yyyy, a common default in digital systems; if the first value exceeds 12, it is interpreted as dd/mm/yyyy. Partial dates (e.g. only a year, or only month plus year) associated to the xsd:gYear or xsd:gYearMonth datatypes. Durations are also standardized to the XSD notation (e.g. 20 minutes is turned into PT20M).

**Entity and Relation Extraction**

* **Location Handling:** Our guidelines for locations prioritize capturing geographical containment. When a text lists a hierarchy of locations (e.g. "Caen, Normandy, France"), we extract each as distinct entity. We then generate isPartOf relations to model their relationship of inclusion (e.g. Caen isPartOf Normandy, Normandy isPartOf France, and Caen isPartOf France). Even though, we take the full string "Caen, Normandy, France" to define a location. For example, Antoine livesAt "Caen, Normandy, France". Finally, definite articles are omitted from place names (e.g., "the Philippines" becomes Philippines).

* **Strict Adherence to Textual Evidence:** Annotations are strictly confined to information explicitly present in the source text, avoiding reliance on external world knowledge. For example, in "Lettria was founded in Paris, France," Paris is typed as Place. However, in "Lettria was
founded in the city of Paris, France,” the explicit mention allows for the more specific type *City*.
This principle ensures that the benchmark evaluates a model’s ability to extract information from
the provided context alone. This rule ensures that the text-2-graph task can be solved relying on
the sole information in the benchmark.

## Entity Scoping

* **Organization names:** Corporate suffixes ("Inc.", "Co.") are preserved as part of the entity name to maintain fidelity to the source text (e.g. *Caterpillar Inc.*).

* **Pronoun Resolution:** We resolve pronouns to their antecedent entity within the extracted triple. For ambiguous pronouns like "which," we employ a heuristic of selecting the immediately preceding noun phrase as the antecedent. For example, in "...beef kway teow which comes from the region of Indonesia," the pronoun "which" is resolved to *beef kway teow*.

* **Multiple Entities:** When a single statement applies to multiple entities, we create a separate triple for each. "Huseyin Butuner and Hilmi Guner designed..." yields two distinct designer relations, one for each person.

## 3.3. The Resulting Benchmark: Curation and Structural Enhancement

The culmination of the re-annotation process, guided by the revised ontology and the new annotation principles, is a benchmark of significantly higher quality and consistency. The resulting dataset comprises a total of 4860 sentences, which correspond to 14882 extracted triples.

In addition to the primary re-annotation, the benchmark underwent a comprehensive data curation and enhancement phase to address artifacts present in the original version and to enrich its structure for more rigorous model evaluation. These post-processing enhancements are detailed as follows:

* **Data Sanitization and Canonicalization:** A systematic normalization process was applied to entity and literal values to ensure uniformity and eliminate parsing inconsistencies. This included several key transformations:

- **Entity Name Normalization:** Underscores used as word separators in entity names were replaced with spaces to form canonical, human-readable identifiers (e.g., "AWH_Engineering_College" was corrected to "AWH Engineering College").

- **Literal Value Cleaning:** Superfluous quotation marks that erroneously encapsulated object values in the original data were removed (e.g., {"obj": "\"Kuttikkattoor\""} was corrected to {"obj": "Kuttikkattoor"}).

- **Numeric Data Typing:** String representations of numbers were parsed into their correct numeric types (e.g., "2000" became 2000). Numerical Values are stripped of punctuation; for example, "18,527" is annotated as 18527 (distinguishing the cases in which the comma was used as thousand or decimal separator).

- **Textual Harmonization:** Spelling inconsistencies and diacritical variations in names were corrected to ensure a true reproduction of what is in the text (e.g., "Hüseyin Bütüner" in the text is kept as it is and not turned into "Huseyin Butuner").

* **Explicit Ontological Typing:** To improve the formal alignment between data instances and the ontology, each triple was enriched with new keys. The subType and objType fields now explicitly declare the ontological class of the subject and the datatype of the object, respectively. This structural addition is critical for enabling type-aware evaluation metrics and enforcing semantic consistency.

* **Corpus and Linguistic Refinement:** The source text corpus itself was subject to a final review. Minor grammatical and punctuation errors were corrected to improve linguistic quality.
The cumulative effect of these enhancements is illustrated in Figure 1 in the Appendix, which presents a side-by-side comparison of a data entry before and after the revision process. Table 3 in the Appendix presents a comparison between the original and new datasets. Text2KGBench-Lettria maintains the same number of sentences as Text2KGBench, while the number of triples varies, showing both additions and reductions respect to Text2KGBench.

# 4. Experimental Evaluation with LLMs

Our study evaluates the performance of contemporary Large Language Models (LLMs) on the Text-to-Knowledge-Graph (Text2KG) task, which involves extracting knowledge graph triples from unstructured text. The evaluation is conducted using the Text2KGBench-Lettria benchmark. We assess two distinct categories of models under different conditions.

First, we assessed a comprehensive suite of proprietary models in a zero-shot setting, where models perform the task without any specific fine-tuning. The evaluated models, grouped by provider, included several from Anthropic, such as the Claude 3 family (Haiku, Sonnet, Opus) [15], the Claude 3.5 series (Haiku, Sonnet V1, Sonnet V2), the Claude 3.7 Sonnet, and the Claude 4 series (Sonnet, Opus). From Google, we evaluated the Gemini 2.0 family (Flash-Lite, Flash, Pro) and the Gemini 2.5 family (Flash-Lite, Flash, Pro) [16]. Our assessment also covered OpenAI's GPT-4.1 series (Full, Mini, Nano) [17] and GPT-4o series (Full, Mini) [18]. Finally, from Mistral AI, we included the Mistral Medium 2505 model<fnref n="1" />.

In parallel, we fine-tuned and subsequently evaluated a selection of prominent open-weights models to gauge their performance after task-specific adaptation. This set comprised Gemma 3 (4B-IT, 12B-IT, 27B-IT) [19]<fnref n="2" />, Mistral Small 3.2 (24B-Instruct)<fnref n="3" />, Phi-4 (14B) [20], and Qwen 3 in several parameter sizes (0.6B, 1.7B, 4B, 8B, 14B, 32B) [21]. We have decided to go with these models because they were at that moment the best instructed pre-trained models on the huggingface leaderboard.

## 4.1. Fine-Tuning Methodology

We employed a Supervised Fine-Tuning (SFT) methodology to adapt the selected Large Language Models (LLMs) for the relation extraction task, utilizing the Unsloth<fnref n="4" /> framework for efficient training. The fine-tuning process is based on Low-Rank Adaptation (LoRA) [22] and involved providing each model with an input prompt containing two components: (1) a natural language sentence and (2) a compact representation of the relevant ontology. To mitigate the verbosity of the standard Turtle syntax and ensure the input fits within the models’ context windows, we adopted a format inspired by Manchester syntax for representing the ontology schema. The target output for the SFT process was a JSON object containing the knowledge graph triples extracted from the sentence, mirroring the ground-truth annotations in our dataset.

To assess model performance under different data conditions, we designed and evaluated three distinct fine-tuning configurations:

**Classic** Models were fine-tuned on the complete, original training dataset. This configuration serves as our performance baseline.

**Extended** This configuration incorporates data augmentation. The original training set was supplemented with synthetic data generated by the Gemini 2.5 Pro model. The objective of this augmentation was to enrich the training data for each ontology, ensuring a number of 500 training examples per ontology, bringing the training set to 9500 examples in total.

**Generalization** This configuration evaluates the models’ zero-shot generalization capabilities to unseen ontologies using a leave-one-out strategy. Models were trained on a dataset comprising

<footnote n="1">¹Model details available at: https://mistral.ai/news/mistral-medium-3</footnote>

<footnote n="2">²Model card: https://huggingface.co/google/gemma-3-12b-it</footnote>

<footnote n="3">³Model card: https://huggingface.co/mistralai/Mistral-Small-3.2-24B-Instruct-2506</footnote>

<footnote n="4">⁴https://unsloth.ai/</footnote>
18 of the 19 ontologies. The held-out ontology (the *City* ontology) was then used exclusively for testing. The final test set for this scenario was composed of all examples (both original training and test splits) associated with the unseen *City* ontology.

All the fine-tuning runs⁵ for each model have been conducted on a Nvidia H100 GPU.

## 4.2. Evaluation

To provide a multifaceted evaluation of our relation extraction approach, we introduce a suite of metrics that extends beyond the traditional F1-score. Our methodology first categorizes the components of the knowledge graph into four distinct types:

**Entities (E)** The classes that serve as the domain and range for object properties, or as the domain for datatype properties.

**Attributes (A)** The literal values that constitute the range of datatype properties.

**Properties (P)** The datatype properties that link entities to attributes.

**Relations (R)** The object properties that link entities to other entities.

Based on this categorization, we assess model performance across six key dimensions:

* **F1-Score:** The macro-averaged F1-score for the correct identification and classification of each extracted entity, attribute, property, and relation.

* **Ontological Fidelity:** A measure to quantify hallucinations, defined as the generation of types, properties, or relations that are not present in the reference ontology.

* **Domain/Range Adherence:** Assesses whether the model's outputs respect the domain and range constraints defined in the ontology for all properties (datatype properties) and relations (object properties). This metric accounts for subclass hierarchies; for instance, if an ontology specifies a domain of *Place* and the model predicts *City*, the prediction is considered valid provided *City* is a subclass of *Place*.

* **Structural Validity:** Measures whether the generated output conforms to the required JSON schema, ensuring it is well-formed and parseable.

* **Latency:** The average inference time in seconds required to generate a response, calculated across all examples in the test set.

* **Cost:** The average monetary cost per query. For proprietary models, this is the API cost. For open-weights models, we estimate the cost based on the hourly price of the required hardware from a cloud provider (e.g., a OVH Cloud instance at 2.80 €/hour).

## 4.3. Performance and Insights

Performance was evaluated using three distinct fine-tuning configurations. The first two configurations were tested on our "full benchmark," a revised and comprehensive version of the new benchmark. The third configuration was subsequently tested on a single ontology in a "generalization" scenario. All experiments involving closed models utilized the most recent, optimized prompt from our internal text-to-graph production framework.
<table><caption>Table 1</caption><thead><tr><th rowspan="2">Run ID</th><th colspan="4">F1</th><th colspan="3">Hallucinations</th><th colspan="2">Respect</th><th rowspan="2">Valid Outputs (%)</th><th rowspan="2">Latency (s)</th><th rowspan="2">Cost ($)</th></tr><tr><th>Entities</th><th>Attributes</th><th>Properties</th><th>Relations</th><th>Types</th><th>Relations</th><th>Properties</th><th>Relations</th><th>Properties</th></tr></thead><tbody><tr><td colspan="14"><strong>Closed - 1-shot</strong></td></tr><tr><td>claude-3.5-haiku</td><td>0.5732</td><td>0.693</td><td>0.6836</td><td>0.5649</td><td>0.0041</td><td>0.0003</td><td>0.0364</td><td>0.9637</td><td>0.9105</td><td>93.44%</td><td>14.8537</td><td>0.0038</td></tr><tr><td>claude-3.5-sonnet-v1</td><td>0.5804</td><td>0.8471</td><td>0.8336</td><td>0.6182</td><td>0.0052</td><td>0.0</td><td>0.0082</td><td>0.978</td><td>0.9826</td><td>97.12%</td><td>14.3797</td><td>0.0139</td></tr><tr><td>claude-3.5-sonnet-v2</td><td>0.6059</td><td>0.8789</td><td>0.8675</td><td>0.6697</td><td>0.0047</td><td>0.0</td><td>0.0065</td><td>0.9886</td><td>0.9935</td><td>99.30%</td><td>15.6948</td><td>0.0136</td></tr><tr><td>claude-3-haiku</td><td>0.5206</td><td>0.3373</td><td>0.3523</td><td>0.4389</td><td>0.0198</td><td>0.0254</td><td>0.2038</td><td>0.8429</td><td>0.6808</td><td>90.66%</td><td>8.6447</td><td>0.0014</td></tr><tr><td>claude-3.7-sonnet</td><td>0.6082</td><td>0.8903</td><td>0.88</td><td>0.6625</td><td>0.004</td><td>0.0</td><td>0.0053</td><td>0.9859</td><td>0.9943</td><td>99.70%</td><td>13.0294</td><td>0.0128</td></tr><tr><td>claude-opus-4</td><td>0.6289</td><td>0.8702</td><td>0.8532</td><td>0.6944</td><td>0.0051</td><td>0.0</td><td>0.0034</td><td>0.9977</td><td>0.9962</td><td>99.20%</td><td>37.4454</td><td>0.1682</td></tr><tr><td>claude-sonnet-4</td><td>0.6487</td><td>0.8657</td><td>0.8498</td><td>0.7126</td><td>0.0011</td><td>0.0</td><td>0.0065</td><td>0.9908</td><td>0.9848</td><td>99.35%</td><td>10.4307</td><td>0.0111</td></tr><tr><td>gemini-2.0-flash-lite</td><td>0.5276</td><td>0.6885</td><td>0.679</td><td>0.5456</td><td>0.0028</td><td>0.0014</td><td>0.0109</td><td>0.9466</td><td>0.9714</td><td>83.95%</td><td>2.3805</td><td>0.0002</td></tr><tr><td>gemini-2.0-flash</td><td>0.3539</td><td>0.4311</td><td>0.4195</td><td>0.3864</td><td>0.0017</td><td>0.0</td><td>0.0137</td><td>0.9626</td><td>0.9799</td><td>57.36%</td><td>1.9308</td><td>0.0004</td></tr><tr><td>gemini-2.5-flash-lite</td><td>0.6014</td><td>0.2542</td><td>0.4930</td><td>0.2335</td><td>0.0088</td><td>0.0221</td><td>0.3439</td><td>0.8553</td><td>0.4993</td><td>97.51%</td><td>1.9086</td><td>0.0010</td></tr><tr><td>gemini-2.5-flash</td><td>0.5501</td><td>0.7463</td><td>0.7339</td><td>0.6062</td><td>0.0055</td><td>0.0</td><td>0.0113</td><td>0.9736</td><td>0.9848</td><td>86.43%</td><td>2.0842</td><td>0.0013</td></tr><tr><td>gemini-2.5-pro</td><td>0.6595</td><td>0.8762</td><td>0.8627</td><td>0.7076</td><td>0.0014</td><td>0.0</td><td>0.0022</td><td>0.9925</td><td>0.9966</td><td>99.80%</td><td>3.9886</td><td>0.005</td></tr><tr><td>gpt-4.1-2025-04-14</td><td>0.6472</td><td>0.8742</td><td>0.863</td><td>0.6565</td><td>0.0014</td><td>0.0004</td><td>0.0146</td><td>0.9798</td><td>0.9843</td><td>97.27%</td><td>3.9289</td><td>0.0058</td></tr><tr><td>gpt-4.1-mini-2025-04-14</td><td>0.6866</td><td>0.8584</td><td>0.8471</td><td>0.6114</td><td>0.0042</td><td>0.0023</td><td>0.0324</td><td>0.9442</td><td>0.9512</td><td>98.86%</td><td>5.9905</td><td>0.0012</td></tr><tr><td>claude-3-opus</td><td>0.6159</td><td>0.7589</td><td>0.7492</td><td>0.6621</td><td>0.0072</td><td>0.0032</td><td>0.0753</td><td>0.9532</td><td>0.8627</td><td>97.12%</td><td>27.2936</td><td>0.068</td></tr><tr><td>gpt-4.1-nano-2025-04-14</td><td>0.4831</td><td>0.5148</td><td>0.4875</td><td>0.1911</td><td>0.0303</td><td>0.1238</td><td>0.4698</td><td>0.5689</td><td>0.4081</td><td>82.75%</td><td>2.5785</td><td>0.0003</td></tr><tr><td>gpt-4o-2024-11-20</td><td>0.6032</td><td>0.7971</td><td>0.7879</td><td>0.6021</td><td>0.0055</td><td>0.0011</td><td>0.0292</td><td>0.9527</td><td>0.9472</td><td>94.14%</td><td>3.5076</td><td>0.0086</td></tr><tr><td>gpt-4o-mini-2024-07-18</td><td>0.5951</td><td>0.4703</td><td>0.5082</td><td>0.3379</td><td>0.0083</td><td>0.0121</td><td>0.2745</td><td>0.8708</td><td>0.666</td><td>91.95%</td><td>9.1931</td><td>0.0005</td></tr><tr><td>mistral-medium-2505</td><td>0.6095</td><td>0.5524</td><td>0.564</td><td>0.6003</td><td>0.0061</td><td>0.0004</td><td>0.1141</td><td>0.9622</td><td>0.7137</td><td>99.11%</td><td>6.4915</td><td>0.0014</td></tr><tr><td>claude-3-sonnet</td><td>0.5869</td><td>0.7303</td><td>0.7246</td><td>0.5583</td><td>0.0088</td><td>0.0026</td><td>0.17</td><td>0.9068</td><td>0.7604</td><td>96.82%</td><td>12.6952</td><td>0.0113</td></tr><tr><td colspan="14"><strong>Open Weights (Finetuned)</strong></td></tr><tr><td>gemma-3-4b-it</td><td>0.8294</td><td>0.9080</td><td>0.8799</td><td>0.7248</td><td>0.0065</td><td>0.0195</td><td>0.0555</td><td>0.9178</td><td>0.8667</td><td>99.35%</td><td>0.0094</td><td>12.0294</td></tr><tr><td>gemma-3-4b-it (ext.)</td><td>0.8329</td><td>0.9344</td><td>0.9089</td><td>0.7647</td><td>0.0069</td><td>0.0124</td><td>0.0290</td><td>0.9438</td><td>0.9143</td><td>99.70%</td><td>0.0092</td><td>11.7466</td></tr><tr><td>gemma-3-12b-it</td><td>0.8606</td><td>0.9211</td><td>0.9001</td><td>0.7942</td><td>0.0066</td><td>0.0118</td><td>0.0335</td><td>0.9570</td><td>0.8904</td><td>99.95%</td><td>0.0129</td><td>16.5804</td></tr><tr><td>gemma-3-12b-it (ext.)</td><td>0.8592</td><td>0.9437</td><td>0.9302</td><td>0.8149</td><td>0.0069</td><td>0.0091</td><td>0.0155</td><td>0.9620</td><td>0.9320</td><td>99.95%</td><td>0.0130</td><td>16.6987</td></tr><tr><td>gemma-3-27b-it</td><td>0.8680</td><td>0.9301</td><td>0.9038</td><td>0.8027</td><td>0.0064</td><td>0.0119</td><td>0.0275</td><td>0.9533</td><td>0.9097</td><td>99.95%</td><td>0.0165</td><td>21.1579</td></tr><tr><td>gemma-3-27b-it (ext.)</td><td>0.8588</td><td>0.9439</td><td>0.9225</td><td>0.8121</td><td>0.0069</td><td>0.0099</td><td>0.0162</td><td>0.9635</td><td>0.9304</td><td>100.00%</td><td>0.0166</td><td>21.2861</td></tr><tr><td>Mistral-Small-3.2</td><td>0.8837</td><td>0.9497</td><td>0.9351</td><td>0.8294</td><td>0.0070</td><td>0.0106</td><td>0.0163</td><td>0.9542</td><td>0.9258</td><td>99.90%</td><td>0.0096</td><td>12.2700</td></tr><tr><td>Mistral-Small-3.2 (ext.)</td><td>0.8764</td><td>0.9474</td><td>0.9287</td><td>0.8307</td><td>0.0072</td><td>0.0089</td><td>0.0245</td><td>0.9641</td><td>0.9185</td><td>100.00%</td><td>0.0096</td><td>12.2708</td></tr><tr><td>phi-4</td><td>0.7420</td><td>0.8451</td><td>0.8112</td><td>0.6359</td><td>0.0161</td><td>0.0329</td><td>0.0549</td><td>0.8946</td><td>0.8548</td><td>93.69%</td><td>0.0080</td><td>10.2748</td></tr><tr><td>phi-4 (ext.)</td><td>0.7656</td><td>0.8802</td><td>0.8432</td><td>0.6810</td><td>0.0091</td><td>0.0178</td><td>0.0364</td><td>0.9300</td><td>0.8838</td><td>96.37%</td><td>0.0079</td><td>10.0703</td></tr><tr><td>Qwen3-0.6B</td><td>0.8272</td><td>0.8980</td><td>0.8653</td><td>0.7059</td><td>0.0118</td><td>0.0222</td><td>0.0425</td><td>0.9150</td><td>0.9100</td><td>99.65%</td><td>0.0064</td><td>8.2034</td></tr><tr><td>Qwen3-0.6B (ext.)</td><td>0.8238</td><td>0.9282</td><td>0.8947</td><td>0.7365</td><td>0.0076</td><td>0.0157</td><td>0.0268</td><td>0.9357</td><td>0.9090</td><td>100.00%</td><td>0.0063</td><td>8.0859</td></tr><tr><td>Qwen3-1.7B</td><td>0.8302</td><td>0.8969</td><td>0.8687</td><td>0.7193</td><td>0.0072</td><td>0.0212</td><td>0.0806</td><td>0.9180</td><td>0.8448</td><td>99.55%</td><td>0.0063</td><td>8.1014</td></tr><tr><td>Qwen3-1.7B (ext.)</td><td>0.8303</td><td>0.9264</td><td>0.8985</td><td>0.7559</td><td>0.0073</td><td>0.0110</td><td>0.0267</td><td>0.9478</td><td>0.9149</td><td>99.65%</td><td>0.0064</td><td>8.1866</td></tr><tr><td>Qwen3-4B</td><td>0.8482</td><td>0.9095</td><td>0.8881</td><td>0.7778</td><td>0.0051</td><td>0.0153</td><td>0.0443</td><td>0.9436</td><td>0.8800</td><td>99.40%</td><td>0.0083</td><td>10.6227</td></tr><tr><td>Qwen3-4B (ext.)</td><td>0.8447</td><td>0.9378</td><td>0.9194</td><td>0.7987</td><td>0.0067</td><td>0.0102</td><td>0.0228</td><td>0.9610</td><td>0.9230</td><td>99.75%</td><td>0.0081</td><td>10.3798</td></tr><tr><td>Qwen3-8B</td><td>0.8512</td><td>0.9137</td><td>0.8875</td><td>0.7758</td><td>0.0069</td><td>0.0143</td><td>0.0411</td><td>0.9512</td><td>0.8960</td><td>99.80%</td><td>0.0084</td><td>10.8296</td></tr><tr><td>Qwen3-8B (ext.)</td><td>0.8412</td><td>0.9351</td><td>0.9190</td><td>0.7949</td><td>0.0067</td><td>0.0095</td><td>0.0254</td><td>0.9583</td><td>0.9228</td><td>99.90%</td><td>0.0082</td><td>10.5739</td></tr><tr><td>Qwen3-14B</td><td>0.8688</td><td>0.9278</td><td>0.9014</td><td>0.8067</td><td>0.0074</td><td>0.0109</td><td>0.0339</td><td>0.9556</td><td>0.9084</td><td>99.80%</td><td>0.0094</td><td>12.0514</td></tr><tr><td>Qwen3-14B (ext.)</td><td>0.8610</td><td>0.9461</td><td>0.9227</td><td>0.8155</td><td>0.0068</td><td>0.0110</td><td>0.0198</td><td>0.9608</td><td>0.9260</td><td>100.00%</td><td>0.0092</td><td>11.7774</td></tr><tr><td>Qwen3-32B</td><td>0.8677</td><td>0.9288</td><td>0.9024</td><td>0.8016</td><td>0.0077</td><td>0.0115</td><td>0.0311</td><td>0.9498</td><td>0.9057</td><td>99.90%</td><td>0.0146</td><td>18.7284</td></tr><tr><td>Qwen3-32B (ext.)</td><td>0.8521</td><td>0.9358</td><td>0.9177</td><td>0.8138</td><td>0.0074</td><td>0.0112</td><td>0.0222</td><td>0.9593</td><td>0.9282</td><td>99.90%</td><td>0.0146</td><td>18.6875</td></tr></tbody></table>

This table compares the performance of various models on the full test set. The first section evaluates closed-source models using a 1-shot prompting strategy. The second section presents results for open-weights models after two finetuning variants: "Classic" (unmarked) and "Extended" (marked with (ext.)).

### 4.3.1. Full Benchmark

**Performance** The most striking finding is the significant performance gap between the two groups. Fine-tuned models operate in a different league, with most achieving an Entity F1 score exceeding 0.80. This underscores the immense power of specialization. The top performer, Mistral-Small-3.2 (ext.), achieved an outstanding Entity F1 of 0.8837, with other models from the Qwen3 and gemma-3 families clustering in the impressive 0.85-0.87 range. In contrast, the proprietary models, which test general-purpose reasoning without task-specific training, top out with an Entity F1 below 0.70. Within this group, a clear performance hierarchy emerges. gemini-2.5-pro stands out as the best allrounder, with consistently high F1 scores across all categories (E=0.6595, A=0.8762, P=0.8627, R=0.7076). Other models act as high-performing specialists: claude-sonnet-4 excels at understanding complex connections with the highest Relations score (R=0.7126), while gpt-4.1-mini-2025-04-14 is best at identifying discrete items (E=0.6866). Meanwhile, models like gemini-2.0-flash and claude-3-haiku struggle with the task's complexity, proving unsuitable for this type of detailed extraction.

<footnote n="5">⁵Fine-Tuning Hyper-Parameters: Lora Rank: 128 Lora Alpha: 512 Batch Size: 1 Gradient Accumulation: 8 Epochs: 3 Warmup Steps: 5 Learning Rate: $2e^{-5}$ Optimisation: AdamW-8bit Weight Decay: 0.01 Learning Rate Scheduler: Linear</footnote>
**Safety and Reliability** Beyond raw performance, fine-tuning proves to be a profound method for ensuring safety and reliability. Nearly all fine-tuned models achieved over 99% validly formatted outputs—with several reaching a perfect 100%—demonstrating that specialization is an exceptionally effective way to guarantee adherence to a specific output format. Furthermore, we observed an "extended effect" in fine-tuned variants: these models often trade a slight dip in Entity F1 for improved scores in other categories and, crucially, lower hallucination rates and better adherence to the ontology. This suggests the -extended process prioritizes overall robustness and safety. Among the proprietary models, the top performers also demonstrate strong reliability. gemini-2.5-pro and claude-opus-4 lead in producing validly formatted outputs (99.80% and 99.20%, respectively) and show superior adherence to the ontology. However, safety is not a given in this category. While models like claude-3.7-sonnet and gemini-2.5-pro boast extremely low hallucination scores, gpt-4.1-nano exhibits a catastrophic failure with a hallucination precision of just 0.4698, making it a high risk for generating false information.

**Efficiency** The efficiency profiles of the two groups present starkly different trade-offs. For the API-based proprietary models, the balance is between performance, latency, and cost-per-call. The gemini-flash models are the fastest, with response times around 2 seconds, while the powerful claude-opus-4 is the slowest at a substantial 37.4 seconds. A similar trade-off exists in cost: gemini-2.0-flash-lite (0.0002¢) is one of the cheapest, whereas claude-opus-4 (0.1682¢) is by far the most expensive, illustrating the classic balance between capability and operational cost. This dynamic shifts entirely with the fine-tuned models, which run on dedicated local hardware. Latencies are astonishingly low, with all models completing the task in under 0.02 seconds—orders of magnitude faster than API calls. The trade-off here is the high, amortized cost of the fine-tuning process and hosting the model on powerful GPU infrastructure. This cost scales directly with model size, making larger models like gemma-3-27b and Qwen3-32B the most expensive to operate.

### 4.3.2. Generalization Benchmark

The Generalization Benchmark results are displayed in Table C in the Appendix.

**Robust Generalization to Unseen Ontologies** The fine-tuned models demonstrate a remarkable capacity for generalization, adeptly applying their learned skills to novel ontologies with only a minimal drop in performance. A direct comparison reveals that the top-performing models maintain their elite status even on unfamiliar schemas. For instance, gemma-3-12b-it achieves an outstanding Entity F1 of 0.8376 on the generalization set, a marginal decrease from its 0.8606 score on the full benchmark. Crucially, this level of performance significantly surpasses that of the best closed-source models on the same generalization task, with gemma-3-12b-it outperforming the top proprietary model, claude-sonnet-4 (0.7829), by a substantial margin. This robustness extends beyond raw F1 scores to safety and reliability; the fine-tuned models maintain their near-zero hallucination rates and high adherence to ontological constraints (e.g., gemma-3-27b-it scores 0.9325 for relations respect), with valid output rates remaining at or near 100%. This indicates that the fine-tuning process instills a deep, transferable understanding of the text-to-graph task structure, creating models that are not only specialized but also highly adaptable and reliable when faced with new, unseen challenges.

### 4.3.3. Lessons Learned

This comprehensive benchmark reveals a clear and instructive dichotomy between specialized, fine-tuned models and general-purpose, proprietary models, offering several key lessons for practitioners.

First, **specialization is paramount for peak performance and reliability**. The fine-tuned open-weights models operate in a separate, higher-performance tier, unambiguously demonstrating that for complex, structured tasks like text-to-graph conversion, task-specific training is the most effective strategy. This superiority is not confined to accuracy metrics like F1 scores; it extends crucially to
output reliability, where fine-tuned models achieve near-perfect adherence to formatting and ontological constraints, effectively eliminating structural errors and minimizing hallucinations.

Second, **effective fine-tuning teaches generalization, not just memorization**. A critical finding is that fine-tuned models maintain their performance advantage even when confronted with entirely unseen ontologies. Their ability to robustly generalize the underlying task structure surpasses even the most advanced proprietary models on the same out-of-domain test set. This proves that the fine-tuning process instills a deep, transferable understanding of the task's logic, making it a viable strategy for building adaptable and scalable systems.

Finally, **the choice between the two approaches hinges on a fundamental trade-off between accessibility and efficiency**. Proprietary models offer an invaluable, zero-setup solution for rapid prototyping and tasks where the overhead of fine-tuning is prohibitive. Within this group, a clear hierarchy exists, with models like *gemini-2.5-pro* and the *claude-4* family providing a strong baseline of general reasoning. However, this convenience comes at the cost of higher latency and a pay-per-call model. In contrast, fine-tuned models represent a strategic investment. While they require significant upfront and ongoing infrastructure costs for training and hosting, they deliver inference speeds that are orders of magnitude faster and are economically superior for high-volume, production-level applications, all while providing unparalleled performance and safety.

# 5. Conclusion and Future Work

In this paper, we introduced Text2KGBench-LettrIA, a rigorously revised benchmark for evaluating ontology-guided Text-to-Knowledge-Graph systems. By systematically overhauling the DBpedia-WebNLG portion of Text2KGBench, we addressed critical limitations in its ontological design, annotation quality, and structural consistency. The resulting benchmark features 19 refined ontologies with enforced hierarchical relationships and strict typing, alongside over 14,000 high-fidelity triples re-annotated under stringent guidelines to ensure textual grounding and reproducibility. This work provides the community with a resource that enables a more precise and nuanced evaluation of model capabilities in structured knowledge extraction.

Our experiments yield a significant finding: smaller, open-weights language models, when properly fine-tuned on our high-quality benchmark, can outperform larger, proprietary models in terms of F1-score for triple extraction. This result underscores the pivotal role that task-specific data quality and model adaptation play in achieving state-of-the-art performance. Nevertheless, our analysis also highlights a persistent challenge: even high-performing models exhibit a tendency to hallucinate or deviate from ontological constraints, indicating that high accuracy on individual components does not guarantee perfect schema adherence.

Building on this work, we identify several key directions for future research.

* **Post-Hoc Alignment:** The prevalence of schema violations and hallucinations, even after supervised fine-tuning (SFT), suggests the need for a subsequent alignment phase. Investigating reinforcement learning-based techniques such as Proximal Policy Optimization (PPO) or Direct Preference Optimization (DPO) could further refine model outputs to improve ontological fidelity.

* **Explainability and Reasoning:** Future work could focus on developing a reasoning layer atop the extraction models. Such a component would not only extract triples but also generate explanations for its predictions, thereby increasing the transparency and trustworthiness of the KG construction process.

* **Context Window Extension:** A current limitation of many open-weights models is their relatively small context window compared to proprietary counterparts. Future experiments should explore methods to extend the effective context size of fine-tuned models, enabling them to process larger and more complex documents and ontologies.
* **Ontology:** The ontologies have only binary relations (they cannot describe complex entities such as event), an improvement would be to create n-ary relations with reification, in order to have more realistic ontologies, and see if the LLMs, even fine-tuned, can properly handle such complex ontologies.

Declaration on Generative AI

During the preparation of this work, the authors used ChatGPT and LeChat by MistralAI in order to: Grammar and spelling check; Paraphrase and reword. After using these tools, the authors reviewed and edited the content as needed and take full responsibility for the publication's content.

Acknowledgments

This work was supported by the French Public Investment Bank (Bpifrance) i-Demo program within
the LettRAGraph project (Grant ID DOS0256163/00).

References

[1] N. Mihindukulasooriya, S. Tiwari, C. F. Enguix, K. Lata, Text2KGBench: A Benchmark for Ontology-Driven Knowledge Graph Generation from Text, in: The Semantic Web – ISWC 2023: 22nd International Semantic Web Conference, Proceedings, Part II, Springer-Verlag, Berlin, Heidelberg, 2023, p. 247–265. doi:10.1007/978-3-031-47243-5_14.

[2] O. Agarwal, H. Ge, S. Shakeri, R. Al-Rfou, Knowledge Graph Based Synthetic Corpus Generation for Knowledge-Enhanced Language Model Pre-training, in: K. Toutanova, A. Rumshisky, L. Zettlemoyer, D. Hakkani-Tur, I. Beltagy, S. Bethard, R. Cotterell, T. Chakraborty, Y. Zhou (Eds.), Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Association for Computational Linguistics, Online, 2021, pp. 3554–3565. doi:10.18653/v1/2021.naac1-main.278.

[3] C. Gardent, A. Shimorina, S. Narayan, L. Perez-Beltrachini, Creating Training Corpora for NLG Micro-Planners, in: R. Barzilay, M.-Y. Kan (Eds.), Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Association for Computational Linguistics, Vancouver, Canada, 2017, pp. 179–188. doi:10.18653/v1/P17-1017.

[4] A. G. Regino, A. Rossanez, R. da Silva Torres, J. C. dos Reis, A Systematic Literature Review on RDF Triple Generation from Natural Language Text, Semantic Web (2025).

[5] S. Tiwari, N. Mihindukulasooriya, F. Osborne, D. Kontokostas, J. D'Souza, M. Kejriwal, et al., Preface for the Third International Workshop on Knowledge Graph Generation from Text, in: 3rd International workshop one knowledge graph generation from text. Data Quality meets Machine Learning and Knowledge Graphs 2024, volume 3747, CEUR-WS, 2024, pp. 1–4.

[6] J. Kalo, T. Nguyen, S. Razniewski, B. Zhang, Preface: Lm-kbc challenge 2024, in: KBC-LM-LM-KBC 2024 Joint proceedings of the KBC-LM workshop and the LM-KBC challenge 2024, CEUR-WS.org, 2024, pp. 1–5.

[7] P.-L. Huguet Cabot, R. Navigli, REBEL: Relation Extraction By End-to-end Language generation, in: Findings of the Association for Computational Linguistics: EMNLP 2021, Association for Computational Linguistics, Punta Cana, Dominican Republic, 2021. URL: https://github.com/Babelscape/rebel/blob/main/docs/EMNLP_2021_REBEL_Camera_Ready_.pdf.

[8] D. Zeng, K. Liu, S. Lai, G. Zhou, J. Zhao, Relation Classification via Convolutional Deep Neural Network, in: J. Tsujii, J. Hajic (Eds.), Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, Dublin City University and Association for Computational Linguistics, Dublin, Ireland, 2014, pp. 2335–2344. URL: https://aclanthology.org/C14-1220/.

[9] Y. Liu, T. Zhang, Z. Liang, H. Ji, D. L. McGuinness, Seq2rdf: An end-to-end application for deriving triples from natural language text, in: Proceedings of the ISWC 2018 Posters & Demonstrations,
Industry and Blue Sky Ideas Tracks co-located with 17th International Semantic Web Conference (ISWC 2018), 2018.

[10] J. Wang, W. Lu, Two are Better than One: Joint Entity and Relation Extraction with Table-Sequence Encoders, in: B. Webber, T. Cohn, Y. He, Y. Liu (Eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Association for Computational Linguistics, Online, 2020, pp. 1706–1721. URL: https://aclanthology.org/2020.emnlp-main.133/. doi:10.18653/v1/2020.emnlp-main.133.

[11] J. Fei, W. Zeng, X. Zhao, X. Li, W. Xiao, Few-Shot Relational Triple Extraction with Perspective Transfer Network, in: Proceedings of the 31st ACM International Conference on Information & Knowledge Management, CIKM '22, Association for Computing Machinery, New York, NY, USA, 2022, p. 488–498. doi:10.1145/3511808.3557323.

[12] L. He, H. Zhang, J. Liu, K. Sun, Q. Zhang, Zero-Shot Relation Triplet Extraction via Knowledge-Driven LLM Synthetic Data Generation, in: D.-S. Huang, Z. Si, C. Zhang (Eds.), Advanced Intelligent Computing Technology and Applications, Springer Nature, Singapore, 2024, pp. 329–340.

[13] Z. Ding, W. Huang, J. Liang, Y. Xiao, D. Yang, Improving Recall of Large Language Models: A Model Collaboration Approach for Relational Triple Extraction, in: N. Calzolari, M.-Y. Kan, V. Hoste, A. Lenci, S. Sakti, N. Xue (Eds.), Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), ELRA and ICCL, Turin, Italy, 2024, pp. 8890–8901. URL: https://aclanthology.org/2024.lrec-main.778/.

[14] A. Ananya, S. Tiwari, N. Mihindukulasooriya, T. Soru, Z. Xu, D. Moussallem, Towards Harnessing Large Language Models as Autonomous Agents for Semantic Triple Extraction from Unstructured Text, in: TEXT2KG 2024: Third International Workshop on Knowledge Graph Generation from Text, Hersonissos, Greece, 2024.

[15] Anthropic, The Claude 3 Model Family: Opus, Sonnet, Haiku, 2024. URL: https://www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/Model_Card_Claude_3.pdf.

[16] G. Comanici, et al., Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities, 2025. URL: https://arxiv.org/abs/2507.06261. arXiv:2507.06261.

[17] OpenAI, et al., GPT-4 Technical Report, 2024. URL: https://arxiv.org/abs/2303.08774. arXiv:2303.08774.

[18] OpenAI, et al., GPT-4o System Card, 2024. URL: https://arxiv.org/abs/2410.21276. arXiv:2410.21276.

[19] Gemma Team, et al., Gemma 3 Technical Report, 2025. URL: https://arxiv.org/abs/2503.19786. arXiv:2503.19786.

[20] M. Abdin, J. Aneja, H. Behl, S. Bubeck, R. Eldan, S. Gunasekar, M. Harrison, R. J. Hewett, M. Java-heripi, P. Kauffmann, J. R. Lee, Y. T. Lee, Y. Li, W. Liu, C. C. T. Mendes, A. Nguyen, E. Price, G. de Rosa, O. Saarikivi, A. Salim, S. Shah, X. Wang, R. Ward, Y. Wu, D. Yu, C. Zhang, Y. Zhang, Phi-4 Technical Report, 2024. URL: https://arxiv.org/abs/2412.08905. arXiv:2412.08905.

[21] A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, C. Zheng, D. Liu, F. Zhou, F. Huang, F. Hu, H. Ge, H. Wei, H. Lin, J. Tang, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Zhou, J. Lin, K. Dang, K. Bao, K. Yang, L. Yu, L. Deng, M. Li, M. Xue, M. Li, P. Zhang, P. Wang, Q. Zhu, R. Men, R. Gao, S. Liu, S. Luo, T. Li, T. Tang, W. Yin, X. Ren, X. Wang, X. Zhang, X. Ren, Y. Fan, Y. Su, Y. Zhang, Y. Zhang, Y. Wan, Y. Liu, Z. Wang, Z. Cui, Z. Zhang, Z. Zhou, Z. Qiu, Qwen3 Technical Report, 2025. URL: https://arxiv.org/abs/2505.09388. arXiv:2505.09388.

[22] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, W. Chen, LoRA: Low-Rank Adaptation of Large Language Models, in: International Conference on Learning Representations, 2022. URL: https://openreview.net/forum?id=nZeVKeeFYf9.
A. Dataset Statistics

<table><caption>Table 2</caption><thead><tr><th rowspan="2">Ontology Name</th><th colspan="3">Text2KGBench-LettrIA</th><th colspan="2">Text2KGBench</th></tr><tr><th>Classes</th><th>Object Prop.</th><th>Datatype Prop.</th><th>Classes</th><th>Object Prop.</th></tr></thead><tbody><tr><td>airport</td><td>11</td><td>15</td><td>5</td><td>13</td><td>39</td></tr><tr><td>artist</td><td>12</td><td>16</td><td>7</td><td>19</td><td>39</td></tr><tr><td>astronaut</td><td>18</td><td>10</td><td>11</td><td>15</td><td>38</td></tr><tr><td>athlete</td><td>10</td><td>18</td><td>9</td><td>14</td><td>37</td></tr><tr><td>building</td><td>9</td><td>11</td><td>12</td><td>14</td><td>38</td></tr><tr><td>celestialbody</td><td>5</td><td>1</td><td>17</td><td>8</td><td>27</td></tr><tr><td>city</td><td>13</td><td>10</td><td>10</td><td>11</td><td>23</td></tr><tr><td>comicscharacter</td><td>8</td><td>8</td><td>4</td><td>10</td><td>18</td></tr><tr><td>company</td><td>9</td><td>13</td><td>6</td><td>10</td><td>28</td></tr><tr><td>film</td><td>5</td><td>10</td><td>5</td><td>18</td><td>44</td></tr><tr><td>food</td><td>12</td><td>13</td><td>2</td><td>12</td><td>24</td></tr><tr><td>meanoftransportation</td><td>12</td><td>20</td><td>28</td><td>20</td><td>68</td></tr><tr><td>monument</td><td>10</td><td>10</td><td>4</td><td>14</td><td>26</td></tr><tr><td>musicalwork</td><td>15</td><td>22</td><td>3</td><td>15</td><td>35</td></tr><tr><td>politician</td><td>17</td><td>25</td><td>9</td><td>19</td><td>40</td></tr><tr><td>scientist</td><td>12</td><td>15</td><td>5</td><td>15</td><td>47</td></tr><tr><td>sportsteam</td><td>9</td><td>12</td><td>3</td><td>14</td><td>24</td></tr><tr><td>university</td><td>11</td><td>16</td><td>11</td><td>15</td><td>46</td></tr><tr><td>writtenwork</td><td>10</td><td>17</td><td>13</td><td>10</td><td>44</td></tr></tbody><tfoot><tr><td><strong>TOTAL</strong></td><td>208</td><td>262</td><td>164</td><td>266</td><td>685</td></tr></tfoot></table>

Comparison of ontology statistics for Text2KGBench-LettrIA and Text2KGBench.
<table><caption>Table 3</caption><thead><tr><th rowspan="2">Ontology</th><th colspan="4">Text2KGBench-LettrIA</th><th colspan="2">Text2KGBench</th></tr><tr><th colspan="4">Sentences</th><th colspan="2">Triples</th></tr><tr><th></th><th>79</th><th>227</th><th>273</th><th>260</th><th>702</th><th>989</th><th>79</th><th>227</th><th>237</th><th>714</th></tr></thead><tbody><tr><td>airport</td><td>84</td><td>302</td><td>198</td><td>256</td><td>896</td><td>638</td><td>84</td><td>302</td><td>252</td><td>896</td></tr><tr><td>artist</td><td>68</td><td>86</td><td>414</td><td>266</td><td>264</td><td>985</td><td>68</td><td>86</td><td>279</td><td>241</td></tr><tr><td>astronaut</td><td>107</td><td>186</td><td>314</td><td>304</td><td>568</td><td>811</td><td>107</td><td>186</td><td>299</td><td>575</td></tr><tr><td>athlete</td><td>103</td><td>172</td><td>328</td><td>276</td><td>593</td><td>956</td><td>103</td><td>172</td><td>309</td><td>588</td></tr><tr><td>building</td><td>72</td><td>122</td><td>378</td><td>203</td><td>329</td><td>885</td><td>72</td><td>122</td><td>223</td><td>373</td></tr><tr><td>celestialbody</td><td>217</td><td>131</td><td>369</td><td>1289</td><td>479</td><td>1038</td><td>217</td><td>131</td><td>651</td><td>398</td></tr><tr><td>city</td><td>36</td><td>66</td><td>434</td><td>92</td><td>165</td><td>934</td><td>36</td><td>66</td><td>107</td><td>215</td></tr><tr><td>comics<br>character</td><td>56</td><td>97</td><td>403</td><td>174</td><td>314</td><td>928</td><td>56</td><td>97</td><td>157</td><td>300</td></tr><tr><td>company</td><td>127</td><td>137</td><td>363</td><td>368</td><td>369</td><td>622</td><td>127</td><td>137</td><td>378</td><td>398</td></tr><tr><td>film</td><td>153</td><td>245</td><td>255</td><td>473</td><td>683</td><td>681</td><td>153</td><td>245</td><td>532</td><td>734</td></tr><tr><td>food</td><td>92</td><td>222</td><td>278</td><td>271</td><td>646</td><td>745</td><td>92</td><td>222</td><td>276</td><td>647</td></tr><tr><td>mean of<br>transportation</td><td>19</td><td>73</td><td>427</td><td>64</td><td>343</td><td>1365</td><td>19</td><td>73</td><td>55</td><td>293</td></tr><tr><td>monument</td><td>209</td><td>81</td><td>419</td><td>842</td><td>285</td><td>912</td><td>209</td><td>81</td><td>604</td><td>221</td></tr><tr><td>musicalwork</td><td>135</td><td>184</td><td>316</td><td>415</td><td>688</td><td>1089</td><td>135</td><td>184</td><td>424</td><td>550</td></tr><tr><td>politician</td><td>149</td><td>110</td><td>390</td><td>387</td><td>259</td><td>559</td><td>149</td><td>110</td><td>411</td><td>300</td></tr><tr><td>scientist</td><td>110</td><td>125</td><td>375</td><td>375</td><td>369</td><td>1294</td><td>110</td><td>125</td><td>401</td><td>375</td></tr><tr><td>sportsteam</td><td>71</td><td>85</td><td>415</td><td>337</td><td>228</td><td>749</td><td>71</td><td>85</td><td>283</td><td>248</td></tr><tr><td>university</td><td>127</td><td>195</td><td>305</td><td>267</td><td>628</td><td>861</td><td>127</td><td>195</td><td>381</td><td>557</td></tr></tbody><tfoot><tr><td>TOTAL</td><td>2014</td><td>2846</td><td>6654</td><td>6919</td><td>8808</td><td>17101</td><td>2014</td><td>2846</td><td>6259</td><td>8623</td></tr></tfoot></table>

Number of sentences and triple per dataset version: test / train / train ext for T2KB-LettrIA and test / train for T2KGBench
## B. Ontology and Annotation Comparison

```json
{
    "id": "ont_1_university_train_37",
    "sent": "The University of Burgundy employs 2900 staff members with 1299 doctoral students",
    "triples": [
        {
            "sub": "University_of_Burgundy",
            "rel": "staff",
            "obj": "2900"
        }, {
            "sub": "University_of_Burgundy",
            "rel": "numberOfDoctoralStudents",
            "obj": "1299"
        }
    ]
}
```

```json
{
    "id": "ont_1_university_train_37",
    "sent": "The University of Burgundy employs 2900 staff members with 1299 doctoral students",
    "triples": [
        {
            "sub": "University of Burgundy",
            "subType": "University",
            "rel": "academicStaffSize",
            "obj": "2900",
            "objType": "integer"
        }, {
            "sub": "University of Burgundy",
            "subType": "University",
            "rel": "numberOfDoctoralStudents",
            "obj": "1299",
            "objType": "integer"
        }
    ]
}
```

<table><caption>Figure 1: Comparison of the same dataset entry in Text2KGBench (left) and Text2KGBench-LettrIA (right).</caption><thead><tr><th>Aspect</th><th>Text2KGBench</th><th>Text2KGBench-LettrIA</th></tr></thead><tbody><tr><td>Domain Coherence</td><td>Included out-of-domain concepts</td><td>Strictly domain-specific concepts.</td></tr><tr><td>Property Semantics</td><td>Ambiguous or overly generic properties</td><td>Properties renamed and specified for clarity.</td></tr><tr><td>Class Structure</td><td>Flat, non-hierarchical</td><td>Hierarchical using subClassOf</td></tr><tr><td>Property Types</td><td>All properties treated as ObjectProperty</td><td>Strict distinction between ObjectProperty and DatatypeProperty with specified domains and ranges.</td></tr><tr><td>URI format</td><td>.../<domain>/<type># where &lt;type&gt; is relations or concepts. E.g. .../ont_19_film/relations#director</td><td>Simplified to .../<domain>#. E.g. .../ont_19_film#director</td></tr><tr><td>Documentation</td><td>Absent</td><td>rdfs:comment for all classes and properties.</td></tr><tr><td>Metadata</td><td>Absent</td><td>Contributor list and owl:versionIRI</td></tr><tr><td>Example</td><td>onto:University a owl:Class ;<br/>rdfs:label "University" .</td><td>onto:University a owl:Class ;<br/>rdfs:subClassOf<br/>onto:AcademicInstitution ;<br/>rdfs:label "University" ;<br/>rdfs:comment "A higher education..." .<br/><br/>onto:AcademicInstitution a owl:Class ;<br/>rdfs:subClassOf onto:Organization ;<br/>rdfs:label "AcademicInstitution" ;<br/>rdfs:comment "An institution for..." .<br/><br/>onto:Organization a owl:Class ;<br/>rdfs:label "Organization" ;<br/>rdfs:comment "A formal structure..." .</td></tr></tbody></table>

<caption>Table 4</caption>

Comparison of the original and revised ontologies, highlighting key structural and semantic enhancements.
C. Generationzation Results

<table><caption>Table 5</caption><thead><tr><th rowspan="2">Run ID</th><th colspan="4">F1</th><th colspan="3">Hallucinations</th><th colspan="2">Respect</th><th rowspan="2">Valid Outputs (%)</th><th rowspan="2">Latency (s)</th><th rowspan="2">Cost ($)</th></tr><tr><th>Entities</th><th>Attributes</th><th>Properties</th><th>Relations</th><th>Types</th><th>Relations</th><th>Properties</th><th>Relations</th><th>Properties</th></tr></thead><tbody><tr><td colspan="14"><strong>Closed - 1-shot</strong></td></tr><tr><td>mistral-medium-2505</td><td>0.7661</td><td>0.7875</td><td>0.796</td><td>0.6444</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.9681</td><td>0.7836</td><td>99.58</td><td>3.2267</td><td>0.0033</td></tr><tr><td>claude-sonnet-4</td><td>0.7829</td><td>0.9509</td><td>0.9283</td><td>0.7179</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.9967</td><td>0.9878</td><td>99.44</td><td>7.5647</td><td>0.0297</td></tr><tr><td>claude-3-opus</td><td>0.7825</td><td>0.9405</td><td>0.9102</td><td>0.7199</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.9836</td><td>0.958</td><td>99.44</td><td>27.9306</td><td>0.1635</td></tr><tr><td>claude-3.5-sonnet-v2</td><td>0.7823</td><td>0.9581</td><td>0.9333</td><td>0.7089</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.9916</td><td>0.9818</td><td>100.00</td><td>14.7003</td><td>0.0319</td></tr><tr><td>claude-3-sonnet</td><td>0.7777</td><td>0.876</td><td>0.8384</td><td>0.6338</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.944</td><td>0.8365</td><td>99.30</td><td>9.9851</td><td>0.0296</td></tr><tr><td>claude-3.7-sonnet</td><td>0.7775</td><td>0.9471</td><td>0.9278</td><td>0.7146</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.9902</td><td>0.9854</td><td>99.72</td><td>11.0482</td><td>0.0311</td></tr><tr><td>gpt-4.1-mini-2025-04-14</td><td>0.7764</td><td>0.8906</td><td>0.8607</td><td>0.6766</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.9839</td><td>0.9764</td><td>99.86</td><td>3.2743</td><td>0.0031</td></tr><tr><td>gemini-2.5-pro</td><td>0.7748</td><td>0.958</td><td>0.9368</td><td>0.7242</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.9913</td><td>0.9881</td><td>99.86</td><td>3.4366</td><td>0.0117</td></tr><tr><td>gpt-4.1-2025-04-14</td><td>0.7731</td><td>0.9193</td><td>0.9013</td><td>0.6773</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.9866</td><td>0.9933</td><td>99.30</td><td>4.351</td><td>0.0154</td></tr><tr><td>gemini-2.5-flash-lite</td><td>0.771</td><td>0.2881</td><td>0.2503</td><td>0.6158</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.9197</td><td>0.4166</td><td>98.33</td><td>2.0679</td><td>0.0011</td></tr><tr><td>mistral-medium-2505</td><td>0.7661</td><td>0.7875</td><td>0.796</td><td>0.6444</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.9681</td><td>0.7836</td><td>99.58</td><td>3.2267</td><td>0.0033</td></tr><tr><td>claude-3.5-sonnet-v1</td><td>0.7539</td><td>0.9277</td><td>0.91</td><td>0.6842</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.987</td><td>0.9815</td><td>97.77</td><td>15.9461</td><td>0.0331</td></tr><tr><td>claude-3.5-haiku</td><td>0.7499</td><td>0.8811</td><td>0.8489</td><td>0.651</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.9729</td><td>0.9201</td><td>97.77</td><td>11.8934</td><td>0.0088</td></tr><tr><td>gpt-4o-2024-11-20</td><td>0.7489</td><td>0.88</td><td>0.8584</td><td>0.6539</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.9766</td><td>0.9754</td><td>95.96</td><td>6.3608</td><td>0.0203</td></tr><tr><td>gemini-2.0-flash-lite</td><td>0.6968</td><td>0.7877</td><td>0.7572</td><td>0.5783</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.9105</td><td>0.9536</td><td>89.54</td><td>2.4511</td><td>0.0006</td></tr><tr><td>claude-3-haiku</td><td>0.6898</td><td>0.4962</td><td>0.4943</td><td>0.5257</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.8894</td><td>0.758</td><td>88.56</td><td>9.7084</td><td>0.0029</td></tr><tr><td>gpt-4o-mini-2024-07-18</td><td>0.68</td><td>0.6632</td><td>0.6585</td><td>0.438</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.9448</td><td>0.7539</td><td>88.42</td><td>7.3224</td><td>0.0012</td></tr><tr><td>gpt-4.1-nano-2025-04-14</td><td>0.6756</td><td>0.6431</td><td>0.6315</td><td>0.2831</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.7622</td><td>0.5021</td><td>91.07</td><td>3.7023</td><td>0.0008</td></tr><tr><td>gemini-2.5-flash</td><td>0.5572</td><td>0.7033</td><td>0.676</td><td>0.5447</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.9921</td><td>0.9824</td><td>74.34</td><td>1.5099</td><td>0.0029</td></tr><tr><td>gemini-2.0-flash</td><td>0.4522</td><td>0.6017</td><td>0.5789</td><td>0.449</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.9843</td><td>0.9837</td><td>64.99</td><td>1.9098</td><td>0.0012</td></tr><tr><td colspan="14"><strong>Open Weights (Finetuned)</strong></td></tr><tr><td>Mistral-Small-3.2</td><td>0.8014</td><td>0.9368</td><td>0.9105</td><td>0.7221</td><td>0.0005</td><td>0.0019</td><td>0.0152</td><td>0.9288</td><td>0.9704</td><td>99.86</td><td>10.8891</td><td>0.0085</td></tr><tr><td>gemma-3-12b-it</td><td>0.8376</td><td>0.9279</td><td>0.8919</td><td>0.7219</td><td>0.0</td><td>0.0</td><td>0.0019</td><td>0.9468</td><td>0.9601</td><td>99.30</td><td>14.911</td><td>0.0116</td></tr><tr><td>gemma-3-27b-it</td><td>0.8372</td><td>0.9315</td><td>0.8901</td><td>0.7061</td><td>0.0</td><td>0.0014</td><td>0.0051</td><td>0.9325</td><td>0.9322</td><td>100.00</td><td>18.9624</td><td>0.0148</td></tr><tr><td>Qwen3-8B</td><td>0.8198</td><td>0.9197</td><td>0.8904</td><td>0.7139</td><td>0.003</td><td>0.005</td><td>0.0025</td><td>0.9629</td><td>0.9789</td><td>100.00</td><td>9.4616</td><td>0.0074</td></tr><tr><td>Qwen3-14B</td><td>0.7943</td><td>0.936</td><td>0.8979</td><td>0.7024</td><td>0.0011</td><td>0.0</td><td>0.0243</td><td>0.9699</td><td>0.9109</td><td>99.72</td><td>10.8778</td><td>0.0085</td></tr><tr><td>Qwen3-1.7B</td><td>0.7827</td><td>0.8951</td><td>0.8552</td><td>0.592</td><td>0.0026</td><td>0.017</td><td>0.0397</td><td>0.8767</td><td>0.8937</td><td>99.02</td><td>7.3623</td><td>0.0057</td></tr><tr><td>Qwen3-4B</td><td>0.7767</td><td>0.9147</td><td>0.8811</td><td>0.6516</td><td>0.0015</td><td>0.006</td><td>0.0228</td><td>0.9054</td><td>0.9221</td><td>98.61</td><td>9.599</td><td>0.0075</td></tr><tr><td>Qwen3-32B</td><td>0.7748</td><td>0.9203</td><td>0.8982</td><td>0.7087</td><td>0.0</td><td>0.0007</td><td>0.0</td><td>0.9512</td><td>0.9272</td><td>99.72</td><td>16.8119</td><td>0.0131</td></tr><tr><td>phi-4</td><td>0.7727</td><td>0.8783</td><td>0.8565</td><td>0.6375</td><td>0.0012</td><td>0.0025</td><td>0.0194</td><td>0.9635</td><td>0.9121</td><td>97.77</td><td>9.0272</td><td>0.007</td></tr><tr><td>Qwen3-0.6B</td><td>0.7659</td><td>0.8768</td><td>0.8329</td><td>0.5129</td><td>0.0044</td><td>0.0372</td><td>0.0601</td><td>0.7991</td><td>0.8892</td><td>99.72</td><td>7.2455</td><td>0.0057</td></tr><tr><td>gemma-3-4b-it</td><td>0.7382</td><td>0.8757</td><td>0.8443</td><td>0.6361</td><td>0.0</td><td>0.0004</td><td>0.0244</td><td>0.9059</td><td>0.9087</td><td>99.30</td><td>10.6816</td><td>0.0083</td></tr></tbody></table>

This table compares the performance of various models on the generalization test set. The first section evaluates closed-source models using a 1-shot prompting strategy. The second section presents results for open-weights fine-tuned models.