DOI: 10.5281/zenodo.15297020
title: OSKGC: A benchmark for Ontology Schema-based Knowledge Graph Construction from text

description: OSKGC is a benchmark dataset designed for knowledge graph construction based on a predefined ontology schema.

Dataset Composition

The OSKGC benchmark consists of three core components:



Text-triple-schema data pairs

Predefined ontology schemas

A unified ontology hierarchy


Based on WebNLG's text-triple pairs, we performed fine-grained type annotation for entities using external knowledge from DBpedia and Wikipedia, and constructed subsumption hierarchies according to DBpedia ontology classes.



Text-Triple-Schema Pairs benchmark/data


OSKGC utilizes text-triple pairs from WebNLG containing 1, 2, and 3 triples, totaling 10,183 text entries. These are divided into three groups by triple count, with each group further categorized into 19 thematic categories (57 total categories). Each text entry is unique, with entity types and relations overlapping across categories, with a total of 207 and 382 respectively. The following is an example entry:

<entry category="3_Airport" id="3_Airport_train_28">
  <text>Allama Iqbal International Airport serves the city of Lahore which is in Pakistan where Nawaz Sharif is the leader.</text>
  <triples>
    <triple>
      <sub>Lahore</sub>
      <rel>country</rel>
      <obj>Pakistan</obj>
    </triple>
    <triple>
      <sub>Allama_Iqbal_International_Airport</sub>
      <rel>serves</rel>
      <obj>Lahore</obj>
    </triple>
    <triple>
      <sub>Pakistan</sub>
      <rel>leader</rel>
      <obj>Nawaz_Sharif</obj>
    </triple>
  </triples>
  <schemas>
    <schema>
      <sub>City</sub>
      <rel>country</rel>
      <obj>Country</obj>
    </schema>
    <schema>
      <sub>Airport</sub>
      <rel>serves</rel>
      <obj>City</obj>
    </schema>
    <schema>
      <sub>Country</sub>
      <rel>leader</rel>
      <obj>PrimeMinister</obj>
    </schema>
  </schemas>
</entry>



<text>: contains textual information

<triples>: defines a set of triples in (subject, relation, object) format

<schemas>: defines corresponding ontology schemas in (subject type, relation, object type) format

Alignment: Triples and schemas maintain strict one-to-one positional correspondence


In our task formulation:



Text serves as model input

Triples and schemas serve as golden labels for:



Triple-level factual accuracy evaluation

Schema-level structural conformity evaluation






Predefined Ontology Schemas benchmark/ontology


For each category, a dedicated ontology schema is constructed to guide knowledge graph construction under the domain. These schemas contain entity types, their relations, and hierarchical structures within each category.



An ontology schema consists of:






Core Structure:



Root-level entity types

Category-specific relations





Hierarchical Structure:



Fine-grained entity types

Connected to core structure via root nodes






Hierarchical Structure benchmark/hierarchy.xml


While relation structures vary by category, all hierarchies follow a standardized framework based on DBpedia ontology classes. This XML file provides the unified subsumption hierarchical structure used throughout OSKGC.

For more details, please refer to our paper and GitHub repository.