Package: linkml-transformer
linkml-transformerLinkML Transformer is a framework for specifying and executing mappings between data models.Features:YAML-based lightweight syntaxPython library for executing mappings on data filesAbility to compile to other frameworks (forthcoming)Derivation of target (implicit) schemas, allowing easy customization of data models (profiling)Simple YAML dictionaries for simple mappingsAutomatic unit conversionUse of subset of Python to specify complex mappingsVisualizations of mappingsMappings are reversible (provided all expressions used are reversible)Status:The transformation data model is not yet fully stable, and may be subject to change.
Not all parts of the model are implemented in the reference Python framework.QuickstartTutorial NotebookGenerated DocsCompliance SuiteBasic ideaGiven an object that conforms to a LinkML schema,e.g.:given_name:Janefamily_name:Doeheight_in_cm:172.0age_in_years:33aliases:[Janey,Janie]Define a mapping in YAML:class_derivations:Individual:populated_from:Personslot_derivations:name:expr:"{given_name}+''+{family_name}"description:Concatenating given and family namesnote this is a bad assumption for names in general,this is just for demonstrationheight_in_meters:populated_from:height_in_cmunit_conversion:target_unit:maliases:populated_from:aliasesstringification:delimiter:'|'Transform the data:name:Jane Doeheight_in_meters:1.72aliases:Janey|JanieInstallation and usageInstallation and command line usage:pip[x]installlinkml-transformercdtests/input/examples/personinfo_basic
linkml-trmap-data\-Ttransform/personinfo-to-agent.transform.yaml\-ssource/personinfo.yaml\data/Container-001.yaml\-ooutput/Translated-001.yamlThe command line has subcommands for:map-data- map data from a source schema to a target schemaderive-schema- derive a target schema from a source schema and a mappinginvert- reverses a mappingcompile- compiles a mapping to another frameworkmarkdown- for generating static sitesgraphviz- for generating visualizationspython- (partial)forthcoming:r2rml, ...DetailsThis repo contains both:A data model for a data modeltransformation languageA reference python implementationThe transformation language is specified in terms of LinkML schemas.
It is intended to be aployglottransformation language, used for
specifying how to map data models independent of underlying representation
(TSVs, JSON/YAML, RDF, SQL Database, ...).Use cases include:ETL and mapping from one data model to anotherDatabase migrations (one version of a schema to another)Creating "profiles"Specifying mappings between different serializations of a model (e.g. OO to Relational)Mapping between normalized/non-redundant forms and denormalized/query-optimized formsTutorialsTutorial Notebook- for Python developersData ModelThe data model for transformations mirrors the data model for schemas:A top levelTransformationSpecificationclass contains:Zero or moreClassDerivationobjects, specifying how to map to a class, containing:Zero or moreSlotDerivationobjects, specifying how to map to a slot, containing:Zero or moreEnumDerivationobjects, specifying how to map permissible values.See thegenerated docsConformance SuiteThe Transformers conformance suite contains a collection of tests for each feature of the language.See:Compliance SuiteRunning the codelinkml-tr--help
Usage:linkml-tr[OPTIONS]COMMAND[ARGS]...CLIforlinkml-transformer.

Options:-v,--verbose-q,--quietTEXT--helpShowthismessageandexit.

Commands:derive-schemaDeriveaschemafromasourceschemaandamapping.map-dataMapdatainasourceschemausingatransformation.map-dataTransforms (maps) data from a source schema to a target schema. This could range from a simple data dictionary mapping
through to a complex mappings.cd tests/input/examples/personinfo_basic
linkml-tr map-data -T transform/personinfo-to-agent.transform.yaml -s source/personinfo.yaml  data/Container-001.yamlderive-schemacd tests/input/examples/personinfo_basic
linkml-tr derive-schema -T transform/personinfo-to-agent.transform.yaml source/personinfo.yamlExamplesSee the tests folder for most up-to-date examplesMapping between two similar data modelsGiven asourceobjectpersons:-id:P:001name:fred bloggsprimary_email:fred.bloggs@example.comage_in_years:33has_familial_relationships:-type:SIBLING_OFrelated_to:P:002current_address:street:1 oak streetaliases:-a-bhas_medical_history:-diagnosis:id:C:001name:c1-diagnosis:id:C:002-id:P:002name:Alison Wuhas_familial_relationships:-type:SIBLING_OFrelated_to:P:001has_medical_history:-diagnosis:id:C:001name:c1 (renamed)organizations:-id:ROR:1name:Acmeand a corresponding schema, consider the case
of mapping to a largely isomorphic schema, with some minor differences:class names are changes (e.g Person to Agent)age is represented as a string, e.g. "33 years"some fields are denormalizedThe mapping may look like:id:my-mappingstitle:my mappingsprefixes:foo:foosource_schema:s1target_schema:s2class_derivations:Container:populated_from:Containerslot_derivations:agents:populated_from:personsAgent:populated_from:Personslot_derivations:## implicitly same name in Agentid:label:populated_from:nameage:expr:"str({age_in_years})+'years'"primary_email:gender:has_familial_relationships:populated_from:has_familial_relationshipsFamilialRelationship:populated_from:FamilialRelationshipslot_derivations:type:related_to:Deriving SchemasFormally a mapping consists of a source schemaS, a target schemaT, and a mappingM.In practice, any of these can bepartiallyspecified and derived from the others.For example:givenS, andT, deriveisomorphic mappingsMbased on shared URIsgivenS andM, deriveT` by applying M to the schemaYou can use thederive-schemacommand to derive a schema from a source schema and a mapping.
This can also be thought of as "profiling" a schema (in the FHIR sense).Seetests/test_mapper/test_schema_mapper.pyfor examplesMeasurements and units-id:P:001height:value:172.0unit:cm<==>-id:P:001height_in_cm:172.0<==>-id:P:001height:"172.0cm"Not supportedAggregation...Why another framework?There are a number of excellent frameworks for performing mapping and data transformations.
The LinkML Transformer framework was born out of a need for a framework that:was not inherently tied to:a particular serialization format (e.g. RDF, JSON, ...)a particular programming language (e.g. Python, Java, ...)a particular database system or database language (e.g. PostgreSQL or SQL or SPARQL)not tied to a particular kind of transformation (e.g. ORM or Tables to RDF)was a natural fit for the LinkML data modeling frameworkwas declarative and easy to perform machine reasoning overis simple for simple use casesIn its current state, this framework is less powerful and expressive than many other frameworks
or methodologies for performing transformations. If you need to perform complex data transformations,
you might be better off using an expressive query language like SPARQL or SQL, or even just coding
transformations directly in a programming language or library like Python or Pandas (but note that
even for the coding use case, the LinkML Transformer framework can be useful as a standard way
ofdocumentingtransformations).Currently the main use case for this framework ismostly isomorphictransformations, with lightweight
manipulations. These lend themselves well to a declarative framework. Uses cases that are a particularly good fit
involve mapping between data-dictionary like standards, with large numbers of metadata elements, where these
elements can often be mapped one-to-one, or with simple manipulations (e.g. unit conversions).The origins lie partly in the SSSOM standard, which is intended as an ultra-simple way of specifying
precise mappings betweenentities(e.g. schema.org Person is the same as DBPedia person). We observed
that many people wanted to extend the language to perform more complex mappings. To address this, we
help a workshop at the Biocuration conference in Padua, Italy in 2022.DiscussionSSSOM paperSSSOM Update 2023SSSOM Complex Mappings Workshop 2023Mapping Data Structures: Challenges and Approaches