Package: d4kms
IntroductionThis read me identifies a set of microservices that implement a range of functionality that allows for the handling of clinical trial data in an automated manner. The use cases addressed areImport of data held in a variety of data formats into study designsExport of data in a number of formatsThe microservices within the system are:Registration AuthorityClinical Recording ModelControlled TerminologyBiomedical ConceptsSDTMFormsStudyThe services are all python services running on FastAPI and uvicorn. They can be run locally using a single shared Neo4j instance or deployed. When deployed each micorservices requires a separate Aura Neo4j instance.Running LocallyUse a single Neo4 instance will all databases for the microservices setup within one instance. Typical names for a devlepoment environment arera-service-devcrm-service-devbc-service-devsdtm-service-devstudy-service-devform-service-devNote: there are no separate data bases for Study Import service or the Study Data Import servicePort NumbersUser Interface ServicesMicroservicePortCT8000BC8001Study8002Form8003SDTM8004Data ServicesMicroservicePortRA8010CRM8011CT8012BC8013Study8014Study Import8015Form8016SDTM8017Study Data Import8018To RunUse themicroservice.commandfile to run. Note this file is mac specific. This will start all of the services and the UIs. There is also a dev_server.sh file in each git repo to run each element individually.DeployedOverviewTo deploy each microservice the following actions should be followed.Create Aura Neo4j instance.Load the data using the data prep utilityDeploy the microserviceCheck all runningNeo4j AuraUse the Aura console to create the database as per the site instructions. Download the file holding the credentials (username and password) for the instanceData LoadOverviewFrom the appropriate data prep projectSetup the virtual environmentSet the environment variables for the credentials of the neo4j instanceSet productionRun the load python fileVirtual EnvironmentRun. ./setup_env.shEnvironment VariablesModify the .production_env file. The file will need to contain the following linesNEO4J_URI=<database uri>
NEO4J_DB_NAME=neo4j
NEO4J_USERNAME=<username>
NEO4J_PASSWORD=<password>
GITHUB=<URL of the github repo main page>The database URI will look something likeneo4j+s://a1bc23d4.databases.neo4j.ioThe githb repo URL will look something likehttps://raw.githubusercontent.com/data4knowledge/ra_prep/main/Set ProductionWe need to tell the load utility to use the production database rather than the local (development) one. This is done via setting thePYTHON_ENVIRONMENTenvironment variable. To so this run. ./set_production.shLoad DataThere will be a python rpogram to load the data. It will be namedstage_<n>_load.py. Run this program by enteringpython stage_2_load.py, here n is 2.The output from the program should look somethig likeDeleting database ...
Database deleted. Load new data ...
https://raw.githubusercontent.com/data4knowledge/ra_prep/main/load_data/node-namespace-1.csv
https://raw.githubusercontent.com/data4knowledge/ra_prep/main/load_data/relationship-manages-1.csv
https://raw.githubusercontent.com/data4knowledge/ra_prep/main/load_data/node-registration_authority-1.csv
<Record file='progress.csv' source='file' format='csv' nodes=9 relationships=7 properties=58 time=4294 rows=0 batchSize=-1 batches=0 done=True data=None>
Load complete. 9 nodes and 7 relationships loaded in 4294 milliseconds.Deploy MicorserviceThe microservices are deployed using the fly.io cloud service.General python instructions are available herehttps://fly.io/docs/languages-and-frameworks/python/Installing the fly command line tool is detailed herehttps://fly.io/docs/hands-on/install-flyctl/deploy the appSet environment variablesEnvironment VariablesSetting environment variables on the server is achieved by using th ecommand line program, either one at a timefly secrets set SUPER_SECRET_KEY=password1234or multiple values, note space delimitedfly secrets set NEO4J_URI=xxx NEO4J_PASSWORD=yyyLoading Studies & DataTo be defined.Building ppackageUse pip to install build and twine. use the following commands to buildpython -m buildand uploadtwine upload dist/*to pypi.org