Package: quadipy
quadipyquadipyis a python package to help transform structured data into RDF graph format.We builtquadipyto enable developers to build a config based ingestion pipeline to an RDF data store (think like FiveTran or Stitch but for RDF).quadipywon't explcitly handle connections to different systems but will allow you to configure the RDF data you want to create from any data source.quadipyleveragesRDFLibto pythonically structure RDF data.The goal with this project is to enable transforming any tabular data structure into graph based RDF data. We go into depthhereon how we have used this config based system to build out our internal knowledge graph at Vouch! You can also check out our talk at KGC'22 on YouTube:Modeling the startup ecosystem using a config based knowledge graph.An example below shows what we mean by translating some tabular data into RDF graph data.For a step by step walk-through of how to do this with more examples, visittutorialsDev SetupRun to set up your dev enviornmentmakesetupUsagefromquadipyimportGraphFormatConfigconfig=GraphFormatConfig.parse_file("path/to/config.json")quads=[config.quadify(record)forrecordinrecords]# records is an Iterator[Dict]quadipycan work with a variety of different data sources as long as what is sent to thequadifymethod is aDictEachQuadcreated has a.to_tuple()method that converts it to a tuple to help facilitate working withRDFLibGraphsfromrdflibimportURIRef,GraphfromquadipyimportQuadquad=Quad(subject=URIRef("Alice"),predicate=URIRef("knows"),obj=URIRef("Bob"),graph=None)g=Graph()g.add(quad.to_tuple())Setting up a GraphFormatConfigThe main value ofquadipycomes from theGraphFormatConfig, which takes in a few parameters to configure the transformation of your data into RDF graph format. We provide configuration examples in theexamplesdirectory to help you get started. The full list of fields that can be configured is described below:fieldrequireddescriptionsource_nameYesA string that is used to describe the source (i.e. "wikipedia" for data from wikipedia)primary_keyYesThis is the key in your data that will be used for the subject of each valuepredicate_mappingYesA mapping where the keys are column names in your data source, and values a nested dict that required apredicate_urikey mapped to the RDF predicate in the target location and an optionalobj_datatypekey that maps to a custom datatype (currently we support [literal,uriref, ordate]). Ifobj_datatypeisn't specified, it will default toliteralsubject_namespaceNoA string prepended to the quad's subject as a namespace, instead of just using the value of theprimary_key. For example, forprimary_key=123andsubject_namespace=wikipediathe values generated wouldNOTbeURIRef("123")butURIRef("wikipedia/123")graph_namespaceNoSimilar tosubject_namespacein that this will assign each fact to a named graph with thegraph_namespace. This is useful to store metadata about fact provenance in named graphs.date_fieldNoThe column in your dataset that the fact's "date" will be pulled from. When specified, the named graph field in each fact will be built from the date. For example ifdate_field=created_atandcreated_at='2021-01-01in the source data the graph field will beURIRef("2021-01-01"). This can can work in conjunction withgraph_namespaceValidate Config filesTo make sure the config files are valid, run the CLI by using the commandquadipyvalidate{path}wherepathcould be the directory for all the config files, e.g.examplesor a single config file e.g.examples/simple.jsonThis script usespydanticvalidator to make sure the config file is a valid JSON file, the required fields are presented and thepredicate_uris are validURIs ("valid" defined by RDFLibhere)