Package: irt-data
Inductive Reasoning with Text (IRT)Table of ContentsInductive Reasoning with Text (IRT)DownloadInstallationLoadingData FormatsGraph DataText DataPyKEEN DatasetPytorch DatasetBring Your Own DataLegacy DataCitationThis code is used to create benchmark datasets as described inOpen-World Knowledge Graph Completion Benchmarks for Knowledge
Discoveryfrom a given knowledge graph (i.e. triple set) and
supplementary text. The two KG's evaluated in the paper (based onFB15k237andCoDEx) are available for
downloadbelow.DownloadWe offer two IRT reference datasets: The first -IRT-FB- is based
onFB15k237and the second -IRT-CDE- utilizesCoDEx. Each dataset offers
knowledge graph triples for theclosed world (cw)andopen world
(ow)split. The ow-split is partitioned into validation and test
data. Each entity of the KG is assigned a set of text contexts of
mentions of that entity.NameDescriptionDownloadIRT-CDEBased on CoDExLinkIRT-FBBased on FB15k237LinkInstallationPython 3.9 is required. We recommendminicondafor
managing Python environments.condacreate-nirtpython=3.9
condaactivateirt
pipinstallirt-dataTherequirements.txtcontains additional packages used for development.LoadingSimply provide a path to an IRT dataset folder. The data is loaded
lazily - that is why the construction is fast, but the first invocation
of.descriptiontakes a while.fromirtimportDatasetdataset=Dataset('path/to/irt-fb')print(dataset.description)IRT DATASET

IRT GRAPH: irt-fb
  nodes: 14541
  edges: 310116 (237 types)
  degree:
    mean 42.65
    median 26

IRT SPLIT
2389 retained concepts

Config:
  seed: 26041992
  ow split: 0.7
  ow train split: 0.5
  relation threshold: 100
  git: 66fe7bd3c934311bdc3b1aa380b7c6c45fd7cd93
  date: 2021-07-21 17:29:04.339909

Closed World - TRAIN:
  owe: 12164
  entities: 12164
  heads: 11562
  tails: 11252
  triples: 238190

Open World - VALID:
  owe: 1558
  entities: 9030
  heads: 6907
  tails: 6987
  triples: 46503

Open World - TEST:
  owe: 819
  entities: 6904
  heads: 4904
  tails: 5127
  triples: 25423

IRT Text (Mode.CLEAN)
  mean contexts: 28.92
  median contexts: 30.00
  mean mentions: 2.84
  median mentions: 2.00Data FormatsThe data in the respective provided dataset folders should be quite
self-explanatory. Each entity and each relation is assigned a unique
integer id (denotede[entity],h[head],t[tail], andr[relation]). There is folder containing the full graph data
(graph/), a folder containing the open-world/closed-world splits
(split/) and the textual data (text/).Graph DataThis concerns both data ingraph/andsplit/. Entity and relation
identifier can be translated with thegraph/entities.txtandgraph/relations.txtrespectively. Triple sets come inh t rorder. Reference code to load graph data:irt.graph.Graph.loadirt.data.dataset.Split.loadText DataThe upstream system that sampled our texts:ecc. All
text comes gzipped and can be opened using the built-in pythongziplibrary. For inspection, you can use thezcat,zless,zgrep,
etc. (at least on unixoid systems ;)) - or extract them usingunzip. Reference code to load text data:irt.data.dataset.Text.loadPyKEEN DatasetFor users ofpykeen. There are two
"views" on the triple sets: closed-world and open-world. Both simply
offer pykeen TriplesFactories with an id-mapping to the IRT
entity-ids.Closed-World:fromirtimportDatasetfromirtimportKeenClosedWorlddataset=Dataset('path/to/dataset')# 'split' is either a single float, a tuple (for an additional# test split) or a triple which must sum to 1kcw=KeenClosedWorld(dataset=dataset,split=.8,seed=1234)print(kcw.description)IRT PYKEEN DATASET
irt-cde

  training triples factory:
    entities: 12091
    relations: 51
    triples: 109910

  validation triples factory:
    entities: 12091
    relations: 51
    triples: 27478It offers.training,.validation, and.testingTriplesFactories,
andirt2keen/keen2irtid-mappings.Open-World:fromirtimportDatasetfromirtimportKeenClosedWorlddataset=Dataset('path/to/dataset')kow=KeenOpenWorld(dataset=ds)print(kow.description)IRT PYKEEN DATASET
irt-cde

  closed world triples factory:
    entities: 12091
    relations: 51
    triples: 137388

  open world validation triples factory:
    entities: 15101
    relations: 46
    triples: 41240

  open world testing triples factory:
    entities: 17050
    relations: 48
    triples: 27577It offers.closed_world,.open_world_valid, and.open_world_testTriplesFactories, andirt2keen/keen2irtid-mappings.Pytorch DatasetFor users ofpytorchand/orpytorch-lightning.We offer atorch.utils.data.Dataset, atorch.utils.data.DataLoaderand apytorch_lightning.DataModule. The dataset abstracts what a
"sample" is and how to collate samples to batches:fromirtimportTorchDataset# given you have loaded a irt.Dataset instance called "dataset"# 'model_name' is one of huggingface.co/modelstorch_dataset=TorchDataset(model_name='bert-base-cased',dataset=dataset,part=dataset.split.closed_world,)# a sample is an entity-to-token-index mapping:torch_dataset[100]# -> Tuple[int, List[int]]# (124, [[101, 1130, ...], ...])# and it offers a collator for batching:batch=TorchDataset.collate_fn([torch_dataset[0],torch_dataset[1]])# batch: Tuple[Tuple[int], torch.Tensor]len(batch[0])# -> 60batch[1].shape# -> 60, 105Note: Only the first invocation is slow, because the tokenizer needs
to run. The tokenized text is saved to the IRT folder undertorch/and re-used from then on.Bring Your Own DataIf you want to utilize this code to create your own
open-world/closed-world-split, you need to either bring your data in a
format readable by the existing code base or extend this code for your
own data model. Seeipynb/graph.split.ipynbfor a step-by-step guide.Legacy DataThis data is used as upstream source or was used in the original
experiments forthe paper. They are left here for documentation
and to allow for reproduction of the original results. You need to go
back to thiscommitin irtm to use the data for model training.NameDescriptionDownloadfb-contexts-v7Original dataset (our text) as used in the paper (all modes, all context sizes)Linkfb-oweOriginal dataset (Wikidata descriptions provided byshah/OWE)Linkfb-db-contexts-v7Our text sampled byeccfor FBLinkcde-contexts-v7Original dataset (our text) as used in the paper (all modes, all contexts sizes)Linkcde-codex.enOriginal dataset (Texts provided bytsafavi/codex)Linkcde-db-contexts-v7Our text sampled byeccfor CDELinkCitationIf this is useful to you, please consider a citation:@inproceedings{hamann2021open,
  title={Open-World Knowledge Graph Completion Benchmarks for Knowledge Discovery},
  author={Hamann, Felix and Ulges, Adrian and Krechel, Dirk and Bergmann, Ralph},
  booktitle={International Conference on Industrial, Engineering and Other Applications of Applied Intelligent Systems},
  pages={252--264},
  year={2021},
  organization={Springer}
}