Package: ioexplorer-dataloader
ioexplorer-dataloaderThis repository contains the code for a command line tool to manage and ingest datasets into a Postgres database, for later use by the IOExplorer web application.Pre-RequisitesThis CLI has three main depencies:dockerandnode, andpython(specifically Python 3).Installation instructions for each can be found below:docker:https://docs.docker.com/install/node:https://nodejs.org/en/python:https://www.python.org/downloads/Oncenodeis installed, you will also need to globally install some packages which are used to interact with the database.npm i -g sequelize sequelize-cli pgWith some environments, you will get a permission error when you attempt to install these packages. There is agood articleon how to fix your environment to avoid these errors.To do a final check to make sure all software is installed, run the following:docker --version
node --version
npm --version
sequelize --version
python --versionNote thatpython --versionshould return something starting with3.Installing the CLIInstalling the CLI:
simply,pip install ioexplorer-dataloader
iodl --helpMake sure you have the most up to date version!
Run the following if you have downloaded in the past:pip install ioexplorer-dataloader --upgradeExample Workflows1. BasicsA workflow to help you get familiar with the basics of the dataloader.Setting environment variablesTo start, we need to set environment variables so that we can interact with a database.
For example, to set up a development database, we can create a file calleddevelopment.envwith the following contents:NODE_ENV=development
IOEXPLORER_MODE=development
IOEXPLORER_DEVELOPMENT_DATABASE_NAME=ioexplorerdb
IOEXPLORER_DEVELOPMENT_DATABASE_HOST=127.0.0.1
IOEXPLORER_DEVELOPMENT_DATABASE_PORT=5432
IOEXPLORER_DEVELOPMENT_DATABASE_USERNAME=root
IOEXPLORER_DEVELOPMENT_DATABASE_PASSWORD=passwordThen callset -a && source development.env.Note:set -awill cause all bash variables which are modified to be exported. This is the equivalent of callingexport <line>for each<line>indevelopment.env.In order to use a production database, you would instead work with a file calledproduction.envwhich would look slightly differentNODE_ENV=production
IOEXPLORER_MODE=production
IOEXPLORER_PRODUCTION_DATABASE_NAME=ioexplorerdb
IOEXPLORER_PRODUCTION_DATABASE_HOST=127.0.0.1
IOEXPLORER_PRODUCTION_DATABASE_PORT=5432
IOEXPLORER_PRODUCTION_DATABASE_USERNAME=root
IOEXPLORER_PRODUCTION_DATABASE_PASSWORD=password
IOEXPLORER_GRAPHQL_URL=http://api:4000/graphqlNote: The differences here are thatDEVELOPMENTis replaced withPRODUCTIONfor many of the environment variables. This is so you can have production and development variables loaded at the same time and easily toggle between the two contexts by settingIOEXPLORER_MODEandNODE_ENV. Also, the environment variableIOEXPLORER_GRAPHQL_URLis added in production, since the processes will be interconnected via a docker network in production rather than via localhost in development. Be sure tounset IOEXPLORER_GRAPHQL_URLin development, or else your graphql client will attempt to connet to the graphql api at the wrong url.Starting a database.Note that the database is not live yet, we only set our environment properly to connect to the database. Start the database by running$ iodl database startThe database should now be started. The database just a docker container running thepostgresimage, so you can see it being run withdocker ps.Opening apsqlshell into a database.Now lets open apsqlshell connected to our newly created database:$ iodl database shell
psql (11.0 (Debian 11.0-1.pgdg90+2))
Type "help" for help.

ioexplorerdb=# \dt
Did not find any relations.The messageDid not find any relations.lets us know that this database is completely empty and schemaless.Applying migrations to our database.TheiodlCLI has a copy of all migrations used to produce the current production version of the IOExplorer database schema. To apply all these migrations run:$ iodl database migrateNow if you open anotherpsqlshell and list the relations, you get the expected:$ iodl database shell
psql (11.0 (Debian 11.0-1.pgdg90+2))
Type "help" for help.

ioepxlorerdb=# \dt
           List of relations
 Schema |     Name      | Type  | Owner
--------+---------------+-------+-------
 public | SequelizeMeta | table | root
 public | cnas          | table | root
 public | datasets      | table | root
 public | fusions       | table | root
 public | mutations     | table | root
 public | samples       | table | root
 public | subjects      | table | root
 public | svs           | table | root
 public | timelines     | table | root
(9 rows)Now our database is ready to get some data.Initializing a datasetcdinto a dataset to upload. Ask Ryan for one if you do not have any.TODO: upload example dataset.The dataset should have the following directory structure:.
├── data_clinical_patient.txt   (R)
├── data_clinical_sample.txt    (R)
├── data_CNA.txt
├── data_fusions.txt
├── data_expression.fpkm.txt
├── data_expression.rld.txt
├── data_expression.raw.txt
├── data_mutations_extended.txt
├── data_SV.txt
└── data_timeline.txtNote:Only the files denoted with an (R) are actually required.We now want toinitializethe dataset. This step willRun some quick validations to make sure the data structure is correct.Collect some meta-information from the user.Write aconfig.yamlfile which stores information about this dataset and helps with ingestion.Run:(dataloader) ryan@galliumos:~/MSK/data/Hugo$ iodl dataset init
INFO: Initializing new dataset!

...
 Some success messages will appear here, or a prompt will ask you if you would like to continue with missing data.
...

? What is the dataset name?  my-dataset
? What is a description of the dataset?  this is a test dataset...
? Enter link to paper.  http://google.com
? Who are you (person uploading data)?  Ryan
SUCCESS: Thanks! I made a file called `config.yaml` in this directory! Check it out and make sure everything looks OK!Ingesting a datasetWith theconfig.yamlfile already formed, ingesting the database is very simple.$ iodl dataset ingestIf there are any problems during ingestion, an error will be thrown and the data that already made it into the database (before the error) will be deleted. This will let you diagnose any problems with the data ingestion and re-attempt ingestion without messing with the state of the database.2. Production on AWSThere are some subtle differences in the above when running on the production AWS server:Instead ofpip, you should usepip-3.6.The environment variables are located in~/ioexplorer/production.env.iodl database shellwill no longer work since the production database is running in a docker swarm. If you would like to shell into the datbase, you can find the id of the database container withdocker container ls.
Look for the line where theIMAGEispostgres:latestand theNAMESis something starting withioexplorer_database. Copy the name string, which should look likeioexplorer_database.1.<a bunch of random characters>. Then, executedocker container exec -it ioexplorer_database.1.<a bunch of random characters> psql -d $IOEXPLORER_PRODUCTION_DATABASE_NAME