Package: transmart-packer
Run data transformation jobs forTranSMART.InstallFirst make virtual environment to install dependencies usingPython 3.7+pipinstalltransmart-packerOr from source:gitclonehttps://github.com/thehyve/transmart-packer.gitcdtransmart-packerpipinstall.Dependenciesa Redis server running on localhost (or updatepacker/config.py)RunningFrom root dir run:redis-servercelery-Apacker.tasksworker--loglevel=infotransmart-packerEnvironment variables:VariableDescriptionTRANSMART_URLThe URL of the TranSMART API serverKEYCLOAK_SERVER_URLKeycloak server URL, e.g.,https://keycloak-dwh-test.thehyve.net/authKEYCLOAK_REALMThe Keycloak realm (default:transmart)KEYCLOAK_CLIENT_IDThe Keycloak client ID (default:transmart-client)KEYCLOAK_OFFLINE_TOKENThe Keycloak offline token.REDIS_URLRedis server URL (default:redis://localhost:6379)DATA_DIRDirectory to write export data (default:/tmp/packer/)LOG_CFGLogging configuration (default:packer/logging.yaml)CLIENT_ORIGIN_URLURLs to restrict cross-origin requests to (CORS) (default:*)An optional variableVERIFY_CERTcan be used to specify the path of a certificate collection file (.pem)
used to verify HTTP requests.KEYCLOAK_OFFLINE_TOKENshould be generated for a system user that has the following roles:realm roleoffline_access– to be able to get the offline token.client roleimpersonationon therealm-managementclient – to support running tranSMART queries on behalf of task users.To get the token, run:KEYCLOAK_CLIENT_ID=transmart-clientSYSTEM_USERNAME=systemSYSTEM_PASSWORD=choose-a-strong-system-password# CHANGE MEKEYCLOAK_SERVER_URL=https://keycloak.example.com/authKEYCLOAK_REALM=examplecurl-f--no-progress-meter\-d"client_id=${KEYCLOAK_CLIENT_ID}"\-d"username=${SYSTEM_USERNAME}"\-d"password=${SYSTEM_PASSWORD}"\-d"grant_type=password"\-d"scope=offline_access"\"${KEYCLOAK_SERVER_URL}/realms/${KEYCLOAK_REALM}/protocol/openid-connect/token"|jq-r'.refresh_token'The value of therefresh_tokenfield in the response is the offline token.To run the stack usingdocker-composefollow the commands below:# Downloads redis image and creates image with project dependencies.docker-composebuild# After build is complete, start via:docker-composeupOn code change the webserver will automatically restart, but the Celery workers will not.
After updating the Celery task logic, you will need to restart the Docker container.UsageAvailable handlers:PathDescriptionGET /jobsList all known jobs for this user.POST /jobs/createCreate a new job by providingjob_typeandjob_parameters, creates the job and returns atask_id.GET/jobs/status/<task_id>Get status details for a specific task.GET/jobs/cancel/<task_id>Cancel scheduled or abort a running task.GET/jobs/data/<task_id>Download the data that this task produced.WS /jobs/subscribeOpen websocket connection to get live updates on job progress.To start the toy job “add” on the localhost machine
make call tohttp://localhost:8999/jobs/create?job_type=add&job_parameters={%22x%22:500,%22y%22:1501}.DevelopmentComponentsOverview of the components of transmart-packer and the interaction with external components.TestingTo run the test suite, we have to start redis-server and celery workers with the commands above.
Then you can run:pythonsetup.pytesttests/csr_observation.json- test data retrieved from TranSMART using the following API call:curl-XPOST-H'Content-type: application/json'-H'Accept: application/json'-d\'{
    "type":"clinical",
    "constraint": {
        "type":"study_name",
        "studyId":"CSR"
    }
}'\'<transmart_api_url>/v2/observations'Current file is created based onclinical test data of python_csr2transmart,
withontology_config.jsonandsources_config.jsonas configuration.Note! Do not change csr_observation.json file manually.ExtendingNew jobs can be added by adding a new Celery function to the jobs folder and adding
the function to the jobs registry. See thepacker/jobs/example.pyto learn how.Existing jobsBasic export jobExport transmart api client observation dataframe to tsv file{"job_type":"basic_export","job_parameters":{"constraint":{"type":"study_name","studyId":"CSR"},"custom_name":"name of the export"}}CSR exportThe Central Subject Registry (CSR) data modelspecific export.
The model contains individual, diagnosis, biosource, biomaterial, radiology and study entities,
following the hierarchy for sample data: patient > diagnosis > biosource > biomaterial.
Studies are orthogonal to samples, i.e., patients are linked to studies independent of samples.
Radiology, same as samples, is linked to patient, but can be also linked to diagnosis (optional).
The entities IDs are first 6 columns of the export file. The rest of the columns are concepts.
Higher level concepts (e.g., Age that is specific to Patient level)
get distributed to all rows specific to lower levels (e.g. Diagnosis)See theCSRtest study as an example orlatest sources datasetthat can be used for e2e testing.{"job_type":"csr_export","job_parameters":{"constraint":{"type":"study_name","studyId":"CSR"},"custom_name":"name of the export","row_filter":{"type":"patient_set","subjectIds":["P2","P6"]}}}where:job_parameters.constraint- anytransmart v2 api constraintor composition of them that used to get data from transmart.job_parameters.custom_name(optional) - name of the export job and the outputtsvfile.job_parameters.row_filter(optional) - anytransmart v2 api constraintor composition of them to fetch keys ([[[[patient],diagnosis], biosource], biomaterial]) that will make it to the end result.
E.g., given theCSRstudy and query above only rows specific toP2andP6patients will end up to the result table such asP2,D2,BS2,BM2, … row.
Please note that keys do not have to be equal in length. A row gets also selected if only part of keys matches. e.g.P1vsP1,D1Adding new entity to CSR data model:When the CSR data model is extended with new sample related entities, the export transformation code
has to be changed as well in order to include a column with the ID of the new entity as one of the identifying columns.In order to do this,packer/table_transformations/csr_transformations.pyfile has to be modified.
TheID_COLUMN_MAPPINGmap needs to be extended with the new dimension name of the new entity
as key and the column name that should appear in the export as value.If the new entity is not a part of the sample hierarchy, but only linked to one or more entities, the
merging logic has to be added intransform_obs_dffunction inpacker/table_transformations/csr_transformations.py(see the example of Radiology and Sample entities).LicenseCopyright © 2019 The Hyve B.V.This program is free software: you can redistribute it and/or modify
it under the terms of the GNU Affero General Public License as
published by the Free Software Foundation, either version 3 of the
License, or (at your option) any later version.This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU Affero General Public License for more details.You should have received a copy of theGNU Affero General Public Licensealong with this program. If not, seehttps://www.gnu.org/licenses/.