Package: nmdc-runtime
A runtime system for NMDC data management and orchestration.Service Statushttp://nmdcstatus.polyneme.xyz/How It Fits Inissuestracks issues related to NMDC, which may necessitate work across multiple repos.nmdc-schemahouses the LinkML schema specification, as well as generated artifacts (e.g. JSON Schema).nmdc-serverhouses code specific to the data portal -- its database, back-end API, and front-end application.workflow_documentationreferences workflow code spread across several repositories, that take source data and produce computed data.This repo (nmdc-runtime)houses code that takes source data and computed data, and transforms it
to broadly accommodate downstream applications such as the data portalmanages execution of the above (i.e., lightweight data transformations) and also
of computationally- and data-intensive workflows performed at other sites,
ensuring that claimed jobs have access to needed configuration and data resources.Data exportsThe NMDC metadata as of 2021-10 is available here:https://drs.microbiomedata.org/ga4gh/drs/v1/objects/sys086d541The link returns aGA4GH DRS API bundle object record, with the NMDC metadata collections (study_set, biosample_set, etc.) as contents, each a DRS API blob object.For example the blob for the study_set collection export, named "study_set.jsonl.gz", is listed with DRS API ID "sys0xsry70". Thus, it is retrievable viahttps://drs.microbiomedata.org/ga4gh/drs/v1/objects/sys0xsry70The returned blob object record listshttps://nmdc-runtime.files.polyneme.xyz/nmdcdb-mongoexport/2021-10-14/study_set.jsonl.gzas the url for an access method.The 2021-10 exports are currently all accessible athttps://nmdc-runtime.files.polyneme.xyz/nmdcdb-mongoexport/2021-10-14/${COLLECTION_NAME}.jsonl.gz, but the DRS API indirection allows these links to change in the future, for mirroring via other URLs, etc. So, the DRS API links should be the links you share.OverviewThe runtime features:Dagsterorchestration:dagit - a web UI to monitor and manage the running system.dagster-daemon - a service that triggers pipeline runs based on time or external state.PostgresSQL database - for storing run history, event logs, and scheduler state.workspace codeCode to run is loaded into a Dagsterworkspace. This code is loaded from
one or more dagsterrepositories. Each Dagsterrepositorymay be run with a different
Python virtual environment if need be, and may be loaded from a local Python file orpip installed from an external source. In our case, each Dagsterrepositoryis simply
loaded from a Python file local to the nmdc-runtime GitHub repository, and all code is
run in the same Python environment.A Dagster repository consists ofsolidsandpipelines,
and optionallyschedulesandsensors.solidsrepresent individual units of computationpipelinesare built up from solidsschedulestrigger recurring pipeline runs based on timesensorstrigger pipeline runs based on external stateEachpipelinecan declare dependencies on any runtimeresourcesor additional
configuration. There are TerminusDB and MongoDBresourcesdefined, as well aspresetconfiguration definitions for both "dev" and "prod"modes. Thepresets tell Dagster to
look to a set of known environment variables to load resources configurations, depending on
themode.ATerminusDBdatabase supporting revision control of schema-validated
data.A MongoDB database supporting write-once, high-throughput internal
data storage by the nmdc-runtime FastAPI instance.AFastAPIservice to interface with the orchestrator and
database, as a hub for data management and workflow automation.Local DevelopmentEnsure Docker (and Docker Compose) are installed; and the Docker engine is running.docker--version
dockercomposeversion
dockerinfoEnsure the permissions of./mongoKeyFileare such that only the file's owner can read or write the file.chmod600./mongoKeyFileEnsure you have a.envfile for the Docker services to source from. You may copy.env.exampleto.env(which is gitignore'd) to get started.cp.env.example.envCreate environment variables in your shell session, based upon the contents of the.envfile.export$(grep-v'^#'.env|xargs)If you are connecting to resources that require an SSH tunnel—for example, a MongoDB server that is only accessible on the NERSC network—set up the SSH tunnel.The following command could be useful to you, either directly or as a template (seeMakefile).makenersc-mongo-tunnelsFinally, spin up the Docker Compose stack.makeup-devDocker Compose is used to start local MongoDB and PostgresSQL (used by Dagster) instances, as well
as a Dagster web server (dagit) and daemon (dagster-daemon).The Dagit web server is viewable athttp://127.0.0.1:3000/.The FastAPI service is viewable athttp://127.0.0.1:8000/-- e.g., rendered documentation athttp://127.0.0.1:8000/redoc/.Local TestingTests can be found intestsand are run with the following commands:On an M1 Mac? May need toexport DOCKER_DEFAULT_PLATFORM=linux/amd64.makeup-test
maketestAs you create Dagster solids and pipelines, add tests intests/to check that your code behaves as
desired and does not break over time.For hints on how to write tests for solids and pipelines in Dagster, see their documentation
tutorial on Testing.Publish to PyPIThis repository contains a GitHub Actions workflow that publishes a Python package toPyPI.You can alsomanuallypublish the Python package to PyPI by issuing the following commands in the root directory of the repository:rm -rf dist
python -m build
twine upload dist/*LinksHere are links related to this repository:Production API server:https://api.microbiomedata.orgPyPI package:https://pypi.org/project/nmdc-runtimeDockerHub image (API server):https://hub.docker.com/r/microbiomedata/nmdc-runtime-fastapiDockerHub image (Dagster):https://hub.docker.com/r/microbiomedata/nmdc-runtime-dagster