Package: semsql
SemSQL: standard SQL views for RDF/OWL ontologiesThis project provides a standard collection of SQL tables/views for ontologies, such that you can make queries like this,
to find all terms starting withAbnormalityinHPO.$sqlitedb/hp.dbsqlite>SELECT*FROMrdfs_label_statementWHEREvalueLIKE'Abnormality of %';stanzasubjectpredicateobjectvaluedatatypelanguageHP:0000002HP:0000002rdfs:labelAbnormality of body heightxsd:stringHP:0000014HP:0000014rdfs:labelAbnormality of the bladderxsd:stringHP:0000022HP:0000022rdfs:labelAbnormality of male internal genitaliaxsd:stringHP:0000032HP:0000032rdfs:labelAbnormality of male external genitaliaxsd:stringReady-made SQLite3 builds can also be downloaded for any ontology inOBO, using URLs such ashttps://s3.amazonaws.com/bbop-sqlite/hp.dbrelation-graphis used to pre-generate tables ofentailed edges. For example,
all is-a and part-of ancestors offingerin Uberon:$sqlitedb/uberon.dbsqlite>SELECT*FROMentailed_edgeWHEREsubject='UBERON:0002389'andpredicateIN('rdfs:subClassOf','BFO:0000050');subject, predicate, objectUBERON:0002389, BFO:0000050, UBERON:0015212UBERON:0002389, BFO:0000050, UBERON:5002389UBERON:0002389, BFO:0000050, UBERON:5002544UBERON:0002389, rdfs:subClassOf, UBERON:0000061UBERON:0002389, rdfs:subClassOf, UBERON:0000465UBERON:0002389, rdfs:subClassOf, UBERON:0000475SQLite provides many advantagesfiles can be downloaded and subsequently queried without network latencycompared to querying a static rdf, owl, or obo file, there is no startup/parse delayrobust and performantexcellent support in many languagesAlthough the focus is on SQLite, this library can also be used for other DBMSs like PostgreSQL, MySQL, Oracle, etcTutorialsSemSQL:notebooks/SemanticSQL-Tutorial.ipynbUsing OAK:part 7 of OAK tutorialInstallationSemSQL comes with a helper Python library. Use of this is optional. To install:pipinstallsemsqlDownload ready-made SQLite databasesPre-generated SQLite database are created weekly for all OBO ontologies and a selection of others (seeontologies.yaml)To download:semsqldownloadobi-oobi.dbOr simply download using URL of the form:https://s3.amazonaws.com/bbop-sqlite/hp.dbAttaching databasesIf you are using sqlite3, then databases can be attached to facilitate cross-database joins.For example, many ontologies use ORCID URIs as the object ofdcterms:contributoranddcterms:creatorstatements, but these are left "dangling". Metadata about these orcids are available in the semsql orcid database instance (derived fromwikidata-orcid-ontology), in theOrcid table.You can useATTACH DATABASEto connect two databases, for example:$sqlite3db/cl.dlsqlite>attach'db/orcid.db'asorcid_db;sqlite>select*fromcontributorinnerjoinorcid_db.orcidon(orcid.id=contributor.object)whereorcid.labellike'Chris%';obo:cl.owl|obo:cl.owl|dcterms:contributor|orcid:0000-0002-6601-2165||||orcid:0000-0002-6601-2165|ChristopherJ.MungallCL:0010001|CL:0010001|dcterms:contributor|orcid:0000-0002-6601-2165||||orcid:0000-0002-6601-2165|ChristopherJ.MungallCL:0010002|CL:0010002|dcterms:contributor|orcid:0000-0002-6601-2165||||orcid:0000-0002-6601-2165|ChristopherJ.MungallCL:0010003|CL:0010003|dcterms:contributor|orcid:0000-0002-6601-2165||||orcid:0000-0002-6601-2165|ChristopherJ.MungallCL:0010004|CL:0010004|dcterms:contributor|orcid:0000-0002-6601-2165||||orcid:0000-0002-6601-2165|ChristopherJ.MungallUBERON:0000093|UBERON:0000093|dcterms:contributor|orcid:0000-0002-6601-2165||||orcid:0000-0002-6601-2165|ChristopherJ.MungallUBERON:0000094|UBERON:0000094|dcterms:contributor|orcid:0000-0002-6601-2165||||orcid:0000-0002-6601-2165|ChristopherJ.MungallUBERON:0000095|UBERON:0000095|dcterms:contributor|orcid:0000-0002-6601-2165||||orcid:0000-0002-6601-2165|ChristopherJ.MungallUBERON:0000179|UBERON:0000179|dcterms:contributor|orcid:0000-0002-6601-2165||||orcid:0000-0002-6601-2165|ChristopherJ.MungallUBERON:0000201|UBERON:0000201|dcterms:contributor|orcid:0000-0002-6601-2165||||orcid:0000-0002-6601-2165|ChristopherJ.MungallUBERON:0000202|UBERON:0000202|dcterms:contributor|orcid:0000-0002-6601-2165||||orcid:0000-0002-6601-2165|ChristopherJ.MungallUBERON:0000203|UBERON:0000203|dcterms:contributor|orcid:0000-0002-6601-2165||||orcid:0000-0002-6601-2165|ChristopherJ.MungallUBERON:0000204|UBERON:0000204|dcterms:contributor|orcid:0000-0002-6601-2165||||orcid:0000-0002-6601-2165|ChristopherJ.MungallCreating a SQLite database from an OWL fileThere are two protocols for doing this:install build dependenciesuse DockerIn either case:The input MUST be in RDF/XML serialization and have the suffix.owl:use robot to convert if format is differentWe are planning to simplify this process in future.1. Build a SQLite database directlyThis requires some basic technical knowledge about how to install things on your machine
and how to put things in your PATH. It does not require Docker.Requirements:rdftab.rsrelation-graphAfter installing these and putting bothrelation-graphandrdftab.rsin your path:semsqlmakefoo.dbThis assumesfoo.owlis in the same folder2. Use DockerThere are two docker images that can be used:ODKsemantic-sqlThe ODK image may lag behinddockerrun-v$PWD:/work-w/work-tilinkml/semantic-sqlsemsqlmakefoo.dbSchemaSeeSchema DocumentationThesource schemais inLinkML- this is then compiled down to SQL Tables and ViewsThe basic idea is as follows:There are a small number of "base tables":statementsprefixentailed_edge- populated by relation-graphAll other tables are actually views (derived tables), and are provided for convenience.ORM LayerA SemSQL relational database can be accessed in exactly the same way as any other SQLdbFor convenience, we provide a Python Object-Relational Mapping (ORM) layer using SQL Alchemy.
This allows for code uchlike the following, which joinsRdfsSubclassOfStatementandexistential restrictions:engine=create_engine(f"sqlite:////path/to/go.db")SessionClass=sessionmaker(bind=engine)session=SessionClass()q=session.query(RdfsSubclassOfStatement)q=q.add_entity(OwlSomeValuesFrom)q=q.join(OwlSomeValuesFrom,RdfsSubclassOfStatement.object==OwlSomeValuesFrom.id)lines=[]forax,exinq.all():line=f'{ax.subject}subClassOf{ex.on_property}SOME{ex.filler}'logging.info(line)lines.append(line)(this example is just for illustration - to do the same thing there is a simpler Edge relation)ApplicationsThe semsql python library is intentionally low level - we recommend using theontology-access-kitFor example:runoak-idb/envo.dbsearcht~biomeYou can also pass in an OWL file and have the sqlite be made on the flyrunoak-isqlite:envo.owlsearcht~biomeEven if using OAK, it can be useful to access SQL tables directly to do complex multi-join queries in a performant way.Optimizationpoetryrunsemsqlview2tableedge--full-index|sqlite3$db/mydb.dbSeeindexesfor some ready-made indexes