Package: candig-ingest
CanDIG-ingestA Python package for batch ingestion and update of clinical and pipeline metadata of candig-server.For more information related to the setup of a candig-server instance, check outhttps://candig-server.readthedocs.io/You may also refer to the SETUP.rst at this repo, fromhttps://github.com/CanDIG/candig-ingest/blob/develop/setup.pyGet startedThis tool is not for standalone use. You must have an existing virtual environment where a candig-server is installed.Once you are in the virtual environment of where your candig-server is, activate it, and runpipinstallcandig-ingestPrepare data for ingestionOnce the package is installed, you may batch ingest or update data. The candig-ingest requires a specially-formatted json file for this purpose.
This page describes the format of the data:https://candig-server.readthedocs.io/en/stable/data.html#clinical-and-pipeline-metadataTo help you get started quicker, we provide a few sample json files that are ready to use, you may retrieve them fromhttps://github.com/CanDIG/candig-ingest/tree/develop/candig/ingest/mock_dataAlternatively, if you need to export data from RedCap APIs, we provide a conversion script that is available fromhttps://github.com/CanDIG/redcap-cloudIngest dataUsage:ingest[-hHelp][-vVersion][-dDescription][--overwrite][-pLoggingPath]<path_to_database><dataset_name><metadata_json>As you can see from above, theingestcommand only has 3 mandatory parameters.If we download a mock data file from the github repo linked above, you will run something like below.You may want to double check if you are in your candig-server’s virtualenv.wgethttps://raw.githubusercontent.com/CanDIG/candig-ingest/develop/candig/ingest/mock_data/clinical_metadata_tier1.jsoningestcandig-example-data/registry.dbmock1clinical_metadata_tier1.json-d"A collection of data from Mars"You may see some warning messages that say “Skipped: Missing 1 or more primary identifiers for record …” if you use the mock data, this is normal.
We designed the mock data to be faulty on purpose. For production data, however, you should not see this message.If you want to add a text description to your dataset, you should use the-dflag, note that the description cannot be updated at this time once
the dataset is created. This is optional, however.Update dataAssume you have data ingested to a database’s dataset already, and would like to update them in batch.If this applies to you, you should specify the–overwriteflag, this will update all records.If you do not see specify this flag, the system will warn you that a record with the same identifier exists.ingestcandig-example-data/registry.dbmock1updated_data.json--overwriteNote that the description of the dataset cannot be changed once it’s created, so a-dflag won’t do anything.Log supportBy default all the actions performed by candig-ingest are logged and stored as log files on the same directory the ingest was called.You may choose another place to store the log files by passing the-pargument every time you run the command:ingestcandig-example-data/registry.dbmock1updated_data.json-p/home/user/Documents/logsQuestions and commentsPlease open an issue here and let us know!