Package: metldata
Metldatametldata - A framework for handling metadata based on ETL, CQRS, and event sourcing.DescriptionScope and Features:Metldata is a framework for handling the entire lifetime of metadata by addressing the
a complex combination of challenges that makes it suitable especially for public
archives for sensitive data:Figure 1| Overview of the combination of challenges during metadata handling.ImmutabilityIt is guaranteed that data entries do not change over time making reproducibility
possible without having to rely on local snapshots.AccessibilityA stable accession is assigned to each resource. Together with the immutability
property, this guarantees that you will always get the same data when querying with the
same accession.Corrections, Improvements, ExtensionsEven though data is stored in an immutable way, the metldata still allows for
corrections, improvements, and extensions of submitted data. This is achieved my not
just storing the current state of a submission but by persisting a version history.
Thereby, modifications are realized by issuing a new version of the submission without
affecting the content of existing versions.TransparencyThe version history not only resolved the conflict between immutability and the need to
evolve and adapt data, it also make the changes transparent to user relying on the data.Multiple RepresentationsOften, the requirements regarding the structure and content of data differs depending
the use case and the audience. Metldata accounts for this by proving a configurable
workflow engine for transforming submitted metadata into multiple representation of that
data.GDPR ComplianceThe GDPR gives data subjects the right to issue a request to delete data. Metldata
complies with this demand. Thereby, only entire versions of a submission can be deleted.
The associated accessions stay available so that user are informed that the associated
data is not available anymore. The guarantees for immutability and stability of
accessions are not violated, however, data might become unavailable.InstallationWe recommend using the provided Docker container.A pre-build version is available atdocker hub:dockerpullghga/metldata:1.1.0Or you can build the container yourself from the./Dockerfile:# Execute in the repo's root dir:dockerbuild-tghga/metldata:1.1.0.For production-ready deployment, we recommend using Kubernetes, however,
for simple use cases, you could execute the service using docker
on a single server:# The entrypoint is preconfigured:dockerrun-p8080:8080ghga/metldata:1.1.0--helpIf you prefer not to use containers, you may install the service from source:# Execute in the repo's root dir:pipinstall.# To run the service:metldata--helpConfigurationParametersThe service requires the following configuration parameters:db_connection_str(string, format: password): MongoDB connection string. Might include credentials. For more information see:https://naiveskill.com/mongodb-connection-string/.Examples:"mongodb://localhost:27017"db_name(string): Name of the database located on the MongoDB server.Examples:"my-database"service_name(string): Default:"metldata".service_instance_id(string): A string that uniquely identifies this instance across all instances of this service. A globally unique Kafka client ID will be created by concatenating the service_name and the service_instance_id.Examples:"germany-bw-instance-001"kafka_servers(array): A list of connection strings to connect to Kafka bootstrap servers.Items(string)Examples:["localhost:9092"]kafka_security_protocol(string): Protocol used to communicate with brokers. Valid values are: PLAINTEXT, SSL. Must be one of:["PLAINTEXT", "SSL"]. Default:"PLAINTEXT".kafka_ssl_cafile(string): Certificate Authority file path containing certificates used to sign broker certificates. If a CA not specified, the default system CA will be used if found by OpenSSL. Default:"".kafka_ssl_certfile(string): Optional filename of client certificate, as well as any CA certificates needed to establish the certificate's authenticity. Default:"".kafka_ssl_keyfile(string): Optional filename containing the client private key. Default:"".kafka_ssl_password(string): Optional password to be used for the client private key. Default:"".primary_artifact_name(string): Name of the artifact from which the information for outgoing change events is derived.Examples:"embedded_public"primary_dataset_name(string): Name of the resource class corresponding to the embedded_dataset slot.Examples:"EmbeddedDataset"resource_change_event_topic(string): Name of the topic used for events informing other services about resource changes, i.e. deletion or insertion.Examples:"searchable_resources"resource_deletion_event_type(string): Type used for events indicating the deletion of a previously existing resource.Examples:"searchable_resource_deleted"resource_upsertion_type(string): Type used for events indicating the upsert of a resource.Examples:"searchable_resource_upserted"dataset_change_event_topic(string): Name of the topic announcing, among other things, the list of files included in a new dataset.Examples:"metadata_datasets"dataset_deletion_type(string): Type used for events announcing a new dataset overview.Examples:"dataset_deleted"dataset_upsertion_type(string): Type used for events announcing a new dataset overview.Examples:"dataset_created"host(string): IP of the host. Default:"127.0.0.1".port(integer): Port to expose the server on the specified host. Default:8080.log_level(string): Controls the verbosity of the log. Must be one of:["critical", "error", "warning", "info", "debug", "trace"]. Default:"info".auto_reload(boolean): A development feature. Set toTrueto automatically reload the server upon code changes. Default:false.workers(integer): Number of workers processes to run. Default:1.api_root_path(string): Root path at which the API is reachable. This is relative to the specified host and port. Default:"/".openapi_url(string): Path to get the openapi specification in JSON format. This is relative to the specified host and port. Default:"/openapi.json".docs_url(string): Path to host the swagger documentation. This is relative to the specified host and port. Default:"/docs".cors_allowed_origins: A list of origins that should be permitted to make cross-origin requests. By default, cross-origin requests are not allowed. You can use ['*'] to allow any origin. Default:null.Any ofarrayItems(string)nullExamples:["https://example.org","https://www.example.org"]cors_allow_credentials: Indicate that cookies should be supported for cross-origin requests. Defaults to False. Also, cors_allowed_origins cannot be set to ['*'] for credentials to be allowed. The origins must be explicitly specified. Default:null.Any ofbooleannullExamples:["https://example.org","https://www.example.org"]cors_allowed_methods: A list of HTTP methods that should be allowed for cross-origin requests. Defaults to ['GET']. You can use ['*'] to allow all standard methods. Default:null.Any ofarrayItems(string)nullExamples:["*"]cors_allowed_headers: A list of HTTP request headers that should be supported for cross-origin requests. Defaults to []. You can use ['*'] to allow all headers. The Accept, Accept-Language, Content-Language and Content-Type headers are always allowed for CORS requests. Default:null.Any ofarrayItems(string)nullExamples:[]artifact_infos(array): Information for artifacts to be queryable via the Artifacts REST API.Items: Refer to#/$defs/ArtifactInfo.loader_token_hashes(array): Hashes of tokens used to authenticate for loading artifact.Items(string)Usage:A template YAML for configurating the service can be found at./example-config.yaml.
Please adapt it, rename it to.metldata.yaml, and place it into one of the following locations:in the current working directory were you are execute the service (on unix:./.metldata.yaml)in your home directory (on unix:~/.metldata.yaml)The config yaml will be automatically parsed by the service.Important: If you are using containers, the locations refer to paths within the container.All parameters mentioned in the./example-config.yamlcould also be set using environment variables or file secrets.For naming the environment variables, just prefix the parameter name withmetldata_,
e.g. for thehostset an environment variable namedmetldata_host(you may use both upper or lower cases, however, it is standard to define all env
variables in upper cases).To using file secrets please refer to thecorresponding sectionof the pydantic documentation.Architecture and Design:The framework uses a combination of ETL, CQRS, and event sourcing. Currently it is
designed to mostly run as a CLI application for managing metadata on the local file
system. However, later, it will be translated into a microservice based-architecture.One Write and Multiple Read RepresentationsInstead of having just a single copy of metadata in a database that
supports all CRUD actions needed by all the different user groups, we
propose to follow the CQRS pattern by having one representation that is
optimized for write operations and multiple use case-specific
representations for querying metadata. Thereby, the write-specific
representation is the source of truth and fuels all read-specific
representations through an ETL process. In the following, the
read-specific representations are also referred to as artifacts.This setup with one write and multiple read representation has the
following advantages:Different subsets of the entire metadata catalog can be prepared
with the needs and the permissions of different user audiences in
mind.It allows for independent scalability of read and write operations.The metadata can be packaged in multiple different formats required
and optimized for different technologies and use cases, such as
indexed searching with ElasticSearch vs. REST or GraphQL queries
supported by MongoDB.Complex write-optimized representations, which are inconvenient for
querying such as event histories, can be used as the source of
truth.Often used metadata aggregations and summary statistics can be
precomputed.Read-specific representations may contain rich annotations that are
not immediately available in the write-specific representation. For
instance, the write-specific representation may only contain one-way
relationships between metadata elements (e.g. a sample might define
ahas_experimentattribute, while an experiment defines nohas_sampleattribute), however, a read-specific representation may
contain two way relationships (e.g. a sample defines ahas_experimentattribute and an experiment defines ahas_sampleattribute).However, there are also disadvantages that are linked to this setup that
should be considered:The write and read representations are only eventually consistent.Adds more complexity than a CRUD setup.Submission-centric Store as The Source of TruthIn the write-specific representation, metadata is packaged into
submissions. Each submission is fully self-contained and linking between
metadata of different submissions is not possible. A submission can have
one of the following statuses:pending - the construction of the submission is in progress, the
submitter may still change its content.in-review - the submitter declared the submission as complete and is
waiting for it to be reviewed, however, both the submitter and the
reviewer can set this submission back to pending to enable further
changes.canceled - the submission was canceled before its completion, its
content was deleted.completed - the submission has been reviewed and approved, the
content of the submission is frozen, and accessions are generated
for all relevant metadata elements.deprecated-prepublication - the submission was deprecated and it
cannot be published anymore, however, its content is not deleted
from the system.emptied-prepublication - the submission was deprecated and its
content was deleted from the system, however, the accessions are not
deleted.published - the submission was made available to other users.deprecated-postpublication - the submission was deprecated and it
should not be used anymore, however, its content stays available to
other users.hidden-postpublication - the submission was deprecated and its
content is hidden from other users but it is not deleted from the
system, the accessions stay available, the submission can be set to
deprecated to make its content available again.emptied-postpublication - the submission was deprecated and its
content was deleted from the system, however, the accessions stay
available.The following status transitions are allowed:pending -> in-reviewpending -> canceledin-review -> completedin-review -> canceledin-review -> pendingcompleted -> publishedcompleted -> deprecated-prepublicationcompleted -> emptied-prepublicationdeprecated-prepublication -> emptied-prepublicationpublished -> deprecated-postpublicationpublished -> hidden-postpublicationpublished -> emptied-postpublicationdeprecated-postpublication -> hidden-postpublicationdeprecated-postpublication -> emptied-postpublicationhidden-postpublication -> deprecated-postpublicationhidden-postpublication -> emptied-postpublicationA deprecated submission may or may not be succeeded by a new submission.
Thereby, the new submission may reuse (a part of) the metadata from the
deprecated submission. The reused metadata including the already
existing accessions is copied over to the new submission so that the
contents of the deprecated submission and the new submission can be
handled independently, for instance, the deprecated submission being
emptied.Event Sourcing to Generate ArtifactsTo implement the ETL processes that generate read-specific artifacts
from the write-specific representation explained above, we propose an
event-sourcing mechanism.The creation and each status change of a given submission (and
accommodating changes to the submission's content) are translated into
events. The events are cumulative and idempotent so you only have to
consume the latest event for a given submission in order to get the
latest state of that submission and a replay of the events will lead to
the same result. Thus, the event history only needs to keep the latest
event for each submission as implemented in the compacted topics offered
by Apache Kafka.Moreover, since submissions are self-contained and do not depend on the
content of other submissions, events of different submissions can be
processed independently.Multiple transformations (as in the ETL pattern) are applied to these
so-called source events to generate altered metadata representations
that are in turn published as events. These derived events can be again
subjected to further transformations.Finally, the derived events are subject to load operations (as in the
ETL pattern) that aggregate the events and bring them into queryable
format (an artifact) that is accessible to users through an API.Metadata Modeling and Model UpdatesMetadata requirements are modeled using LinkML. Thereby, the metadata
model should take the whole metadata lifecycle into account so that it
can be used to validate metadata before and after the submission as well
as for all derived artifacts.Updates to the metadata model are classified into minor and major ones.
For minor updates, existing submissions are automatically migrated. The
submission always stores metadata together with the used metadata model.
The migration is realized through scripts that migrate metadata from an
old version to a newer version. Multiple migration scripts may be
combined to obtain a metadata representation that complies with the
newest version. The migration can be implemented as a transformation
that is applied to the source events as explained above.DevelopmentFor setting up the development environment, we rely on thedevcontainer featureof VS Code
in combination with Docker Compose.To use it, you have to have Docker Compose as well as VS Code with its "Remote - Containers"
extension (ms-vscode-remote.remote-containers) installed.
Then open this repository in VS Code and run the commandRemote-Containers: Reopen in Containerfrom the VS Code "Command Palette".This will give you a full-fledged, pre-configured development environment including:infrastructural dependencies of the service (databases, etc.)all relevant VS Code extensions pre-installedpre-configured linting and auto-formattinga pre-configured debuggerautomatic license-header insertionMoreover, inside the devcontainer, a convenience commandsdev_installis available.
It installs the service with all development dependencies, installs pre-commit.The installation is performed automatically when you build the devcontainer. However,
if you update dependencies in the./pyproject.tomlor the./requirements-dev.txt, please run it again.LicenseThis repository is free to use and modify according to theApache 2.0 License.README GenerationThis README file is auto-generated, please seereadme_generation.mdfor details.