Package: ampscz-lochness
Lochness: Sync data from all over the cloud to a local directoryLochness is a data management tool designed to periodically poll and
download data from various data archives into a local directory. This
is often referred to as building a "data lake" (hence the name).Out of the box there is support for pulling data down from Beiwe, XNAT,
REDCap, Dropbox, external hard drives, and more. Extending Lochness to
support new services is also a fairly simple process.Table of contentsInstallationQuick setup from a templateDocumentationInstallationJust usepippip install lochnessFor most recent DPACC-lochnesspip install git+https://github.com/PREDICT-DPACC/lochnessFor debuggingcd ~
git clone https://github.com/PREDICT-DPACC/lochness
pip install -r ~/lochness/requirements.txt

export PATH=${PATH}:~/lochness/scripts  # add to ~/.bashrc
export PYTHONPATH=${PYTHONPATH}:~/lochness  # add to ~/.bashrcRunning testCopy the token template, and add the information for each module.cd lochness/tests
cp token_template_for_test_template.csv token_template_for_test.csvRun testbash run_test.shSetup from a templateCreating the templateSetting up lochness from scratch could be slightly confusing in the beginning.
Try using thelochness_create_template.pyto create a starting point.Create an example template to easily structure the lochness system# ProNETlochness_create_template.py\--outdir/data/lochness_root\--studiesPronetLAPronetSLPronetWU\--sourcesredcapxnatboxmindlamp\--emailkevincho@bwh.harvard.edu\--poll_interval43200\--ssh_hosterisone.partners.org\--ssh_userkc244\--lochness_sync_send\--s3# PRESCIENTlochness_create_template.py\--outdir/data/lochness_root\--studiesPrescientADPrescientMEPrescientPE\--sourcesRPMSmediafluxmindlamp\--emailkevincho@bwh.harvard.edu\--poll_interval43200\--ssh_hosterisone.partners.org\--ssh_userkc244\--lochness_sync_send\--s3# For more options: lochness_create_template.py -hMaking edits to the templateRunning one of the commands above will create the structure below/data/lochness_root/
├──1_encrypt_command.sh
├──2_sync_command.sh
├──PHOENIX
│├──GENERAL
││├──PronetLA
│││└──PronetLA_metadata.csv
││├──PronetSL
│││└──PronetSL_metadata.csv
││└──PronetWU
││└──PronetWU_metadata.csv
│└──PROTECTED
│├──PronetLA
│├──PronetSL
│└──PronetWU
├──config.yml
├──lochness.json
└──pii_convert.csvChange information inconfig.ymlandlochness.jsonas needed.Either manually update thePHOENIX/GENERAL/*/*_metadata.csvor
amend the field names in REDCap / RPMS sources correctly for lochness to
automatically update the metadata files.Currently, lochness initializes the metadata using the following field names
in REDCap and RPMS.chric_subject_id: the record ID field namethis field name must be in the REDCap or RPMS repository for the metadata
to be updated by lochness.chric_consent_date: the field name of the consent datethis field name must be in the REDCap or RPMS repository for the metadata
to be updated by lochness.beiwe_id: the field name of the BEIWE ID.xnat_id: the field name of the XNAT ID.dropbox_id: the field name of the Dropbox ID.box_id: the field name of the Box ID.mediaflux_id: the field name of the Mediaflux ID.mindlamp_id: the field name of the Mindlamp ID.daris_id: the field name of the DaRIS ID.rpms_id: the field name of the RPMS ID.Encrypt thelochness.jsonby runningcd/data/lochness_root
bash1_encrypt_command.shThis encryption step creates a copy of encrypted keyrings to/data/lochness_root/.lochness.enc. To protect the sensitive keyring
information in json, remove thelochness.jsonafter running the encryption.You can still extract keyring structure without sensitive information by runninglochness_check_config.py -ke /data/lochness_root/.lochness.encSet up REDCap Data Entry Trigger if using REDCap. Please see below
"REDCap Data Entry Trigger capture" section.Edit Personally identifiable information mapping table. Please seee below
"Personally identifiable information removal from REDCap and RPMS data"/data/lochness_root/pii_convert.csvRun thesync.pyor use the example command in2_sync_command.shbash 2_sync_command.shTo uselochness_to_lochnesstransfer throughaws s3Set up s3 bucketInstall aws CLIConfigure CLI with your s3 bucket information$ aws configureAdd your AWS information toconfig.ymlAWS_BUCKET_NAME: ampscz-dev
AWS_BUCKET_ROOT: TEST_PHOENIX_ROOTREDCap Data Entry Trigger captureIf your sources include REDCap and you would like to configure lochness to
only pull new REDCap data, "Data Entry Trigger" needs to be set up in REDCap.In REDCap,"Project Setup""Enable optional modules and customizations""Additional customizations"Check "Data Entry Trigger" and give address of the server including the port number e.g.http://pnl-t55-7.partners.org:9999In order to use this functionality, the server where lochness is installed
should be able to receieve HTTP POST signal from REDCap server. Which means it
has to be eitherlochness server is inside the same firewall as REDCap server.
Orlochness server has a open port that could listen to the REDCap POST signal.After setting the "Data Entry Trigger" on REDCap settings, run below to update
the/data/data_entry_trigger_db.csvreal-time# please specify the same port defined in the REDCap settings
listen_to_redcap.py --database_csv /data/data_entry_trigger_db.csv \
                    --port 9999It would be useful to runlisten_to_redcap.pyin background, maybe inside agnu screenso it runs continuously without interference.Personally identifiable information removal from REDCap and RPMS dataA path of csv file can be provided, which has information about how to process
each PII fields.For example/data/personally_identifiable_process_mappings.csvpii_label_string | process
-----------------|---------------
address          | remove
date             | change_date
phone_number     | random_number
patient_name     | random_string
subject_name     | replace_with_subject_idAny value from the field, with names that match topii_label_stringrows,
the labelledPII processing methodwill be used to process the raw values
to remove or replace the PIIs.DocumentationYou can find all the documentation you will ever needhere