Package: sample-metadata-ms
Sample MetadataThe sample-metadata system is database that storesde-identifiedmetadata.There are three components to the sample-metadata system:System-versioned MariaDB database,Python web API to manage permissions, and store frequently used queries,An installable python library that wraps the Python web API (using OpenAPI generator)Every resource in sample-metadata belongs to a project. All resources are access
controlled through membership of the google groups:$dataset-sample-metadata-main-{read,write}. Note that members of google-groups
are cached in a secret as group-membership identity checks are slow.StructureSample IDsIn an effort to reduce our dependency on potentially mutable external sample IDs with inconsistent format,
the sample-metadata server generates an internal sample id for every sample. Internally they're an
incrementing integer, but they're transformed externally to have a prefix, and checksum - this allows durability
when transcribing sample IDs to reduce mistypes, and allow to quickly check whether a sample ID is valid.NB: The prefix and checksums are modified per environment (production, development, local) to avoid duplicates from these environments.For example, let's consider the production environment which uses the prefix ofCPGand a checksum offset of 0:A sample is given the internal ID12345, we calculate the Luhn checksum to be5(with no offset applied).
We can then concatenate the results, for the final sample ID to beCPG123455.Reporting sexTo avoid ambiguity in reporting of gender, sex and karyotype - the sample metadata system
stores these values separately on theparticipantas:reported_gender(string, expectedmale|female|other values)reported_sex(follows pedigree convention:unknown=0 | null,male=1,female=2)inferred_karyotype(string, eg:XX|XY|other karyotypes)If you import a pedigree, the sex value is written to thereported_sexattribute.Local develompent of SMThe recommended way to develop the sample-metadata system is to run a local copy of SM.There have been some reported issues of running a local SM environment on an M1 mac.You can run MariaDB with a locally installed docker, or from within a docker container.
You can configure the MariaDB connection with environment variables.Creating the environmentDependencies for thesample-metadataAPI package are listed insetup.py.
Additional dev requirements are listed inrequirements-dev.txt, and packages for
the sever-side code are listed inrequirements.txt.To create the full dev environment, run:virtualenvvenvsourcevenv/bin/activate
pipinstall-rrequirements.txt
pipinstall-rrequirements-dev.txt
pipinstall--editable.Default DB set-upThese are the default values for the SM database connection.
Please alter them if you use any different values when setting up the database.exportSM_DEV_DB_USER=rootexportSM_DEV_DB_PASSWORD=# empty passwordexportSM_DEV_DB_HOST=127.0.0.1exportSM_DEV_DB_PORT=3306# default mariadb portCreate the database in MariaDB (by default, we call itsm_dev):
If you use a different databse name also set the followingexportSM_DEV_DB_NAME=sm_database_nameSample-metadata stores all metadata in one database (previously: one database per project).mysql-uroot--execute'CREATE DATABASE sm_dev'Download themariadb-java-clientand create the schema using liquibase:pushddb/
wgethttps://repo1.maven.org/maven2/org/mariadb/jdbc/mariadb-java-client/3.0.3/mariadb-java-client-3.0.3.jar
liquibase\--changeLogFileproject.xml\--urljdbc:mariadb://localhost/sm_dev\--driverorg.mariadb.jdbc.Driver\--classpathmariadb-java-client-3.0.3.jar\--usernameroot\updatepopdUsing Maria DB docker imagePull mariadb imagedockerpullmariadbRun a mariadb container that will server your database.-p 3307:3306remaps the port to 3307 in case if you local MySQL is already using 3306dockerstopmysql-p3307# stop and remove if the container already existsdockerrmmysql-p3307# run with an empty root passworddockerrun-p3307:3306--namemysql-p3307-eMYSQL_ALLOW_EMPTY_PASSWORD=true-dmariadbmysql--host=127.0.0.1--port=3307-uroot-e'CREATE DATABASE sm_dev;'mysql--host=127.0.0.1--port=3307-uroot-e'show databases;'Go into thedb/subdirectory, download themariadb-java-clientand create the schema using liquibase:pushddb/
wgethttps://repo1.maven.org/maven2/org/mariadb/jdbc/mariadb-java-client/3.0.3/mariadb-java-client-3.0.3.jar
liquibase\--changeLogFileproject.xml\--urljdbc:mariadb://127.0.0.1:3307/sm_dev\--driverorg.mariadb.jdbc.Driver\--classpathmariadb-java-client-3.0.3.jar\--usernameroot\updatepopdFinally, make sure you configure the server (making use of the environment variables) to point it to your local Maria DB serverexportSM_DEV_DB_PORT=3307Running the serverYou'll want to set the following environment variables (permanently) in your
local development environment.# ensures the SWAGGER page (localhost:8000/docs) points to your local environmentexportSM_ENVIRONMENT=LOCAL# skips permission checks in your local environmentexportSM_ALLOWALLACCESS=true# start the serverpython3-mapi.server# OR# uvicorn --port 8000 --host 0.0.0.0 api.server:appIn a different terminal, execute the following
request to create a new project called 'dev'curl-X'PUT'\'http://localhost:8000/api/v1/project/?name=dev&dataset=dev&gcp_id=dev&create_test_project=false'\-H'accept: application/json'\-H"Authorization: Bearer$(gcloudauthprint-identity-token)"Quickstart: Generate and install the installable APIIt's best to do this with an already running server:python3regenerate_api.py\&&pipinstall.Debugging the server in VSCodeVSCode allows you to debug python modules, we could debug the web API atapi/server.pyby considering the followinglaunch.json:{"version":"0.2.0","configurations":[{"name":"API server","type":"python","request":"launch","module":"api.server"}]}We could now place breakpoints on the sample route (ie:api/routes/sample.py), and debug requests as they come in.Developing the UI# Ensure you have started sm locally on your computer already, then in another tab open the UI.# This will automatically proxy request to the server.cdweb
npminstall
npmstartUnauthenticated accessYou'll want to set theSM_LOCALONLY_DEFAULTUSERenvironment variable along withALLOWALLACCESSto allow access to a local sample-metadata server without providing a bearer token. This will allow you to test the front-end components that access data. This happens automatically on the production instance through the Google identity-aware-proxy.exportSM_ALLOWALLACCESS=1exportSM_LOCALONLY_DEFAULTUSER=$(whoami)OpenAPI and SwaggerThe Web API usesapispecwith OpenAPI3 annotations on each route to describe interactions with the server. We can generate a swagger UI and an installable
python module based on these annotations.Some handy links:OpenAPI specificationDescribing parametersDescribing request bodyMedia typesThe web API exposes this schema in two ways:Swagger UI:http://localhost:8000/docsYou can use this to construct requests to the serverMake sure you fill in the Bearer token (at the top right )OpenAPI schema:http://localhost:8000/schema.jsonReturns a JSON with the full OpenAPI 3 compliant schema.You could put this into theSwagger editorto see the same "Swagger UI" that/api/docsexposes.We generate the sample_metadata installable Python API based on this schema.Generating the installable APIThe installable API is automatically generated through thepackage.ymlGitHub action and uploaded to PyPI.To generate the python api you'll need to install openapi generator v5.x.xTo install a specific version of the openapi-generator dow the following:npminstall@openapitools/openapi-generator-cli-g
openapi-generator-cliversion-managerset5.3.0Then set your environment variable OPENAPI_COMMAND to the following.
You can also add an alias to your ~/.bash_profile or equivalent for running in the
terminal.exportOPENAPI_COMMAND="npx @openapitools/openapi-generator-cli"aliasopenapi-generator="npx @openapitools/openapi-generator-cli"You could generate the installable API and install it with pip by running:# this will start the api.server, so make sure you have the dependencies installed,pythonregenerate_api.py\&&pipinstall.Or you can build the docker file, and specify that# SM_DOCKER is a known env variable to regenerate_api.pyexportSM_DOCKER="cpg/sample-metadata-server:dev"dockerbuild--build-argSM_ENVIRONMENT=local-t$SM_DOCKER-fdeploy/api/Dockerfile.
pythonregenerate_apy.pyDeploymentThe sample-metadata serverYou'll want to complete the following steps:Ensure there is a database created for each project (with the database name being the project),Ensure there are secrets inprojects/sample_metadata/secrets/databases/versions/latest, that's an array of objects with keysdbname, host, port, username, password.Ensuregoogle-cloudwas installedexportSM_ENVIRONMENT='PRODUCTION'# OR, point to the dev instance withexportSM_ENVIRONMENT='DEVELOPMENT'