Package: nrc-ngs-dl
NRC-LIMS-dataDownloaderDescriptionNRC-LIMS-Datadownloader is a software written in Python. This software explores the NRC-LIMS website, downloads all the sequence files, and keeps the meta data of all the sequences in a sqlite database.The list of the tasks performed by the software:Scrapes the NRC-LIMS website to get a list of all the completed sequence runs and all information related to sequence runs and sequence files.Obtains new runs that were not been previously downloaded or re-processed/modified runs by checking each sequence run against the database.Download each new/re-processed run's data and subsequently unzips the file to obtain demultiplexed fastq filesRenames each fastq file to the submitted sample name from the sequencing run information page.Generates a SHA256 code for each fastq file and gzips the fileInserts information about newly downlaoded runs and files into databaseRequirementsPython 2.7VirtualEnvGNU MakeDeployment ProceduresCreate and start the virtual enviromentcd path/to/your/foldervirtualenv -p /path/to/python2.7 venvsource venv/bin/activateInstall the program and all the dependenciespip install nrc_ngs_dlCopy the sample configuration fileconfig.ini.sampletoconfig.iniand provide the required settingscp venv/bin/config.ini.sample config.inivim config.iniRun the programcd path/to/your/foldersource venv/bin/activatelims_downloader -c config.iniSet up the HCRON serviceGet the permission to access hcron1.science.gc.ca by opening an IT centre ticket with message:HPC Dorval - Supercomputing - DC000131Please register my account   chz001   to use hcron on:hcron1.science.gc.caSetup Passwordless Loginhttps://portal.science.gc.ca/confluence/display/SCIDOCS/SSH+Login+without+a+Passwordmkdir -p ~/.sshchmod -R 700 ~/.sshcd ~/.sshssh-keygen -q -t rsacat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keyCreate the home for your hcron events:bash-4.1$ mkdir -p  ~/.hcron/hcron1.science.gc.ca/eventsCreate a hcron event file (file downloader as an example) :bash-4.1$ cd ~/.hcron/hcron1.science.gc.ca/eventsbash-4.1$ hcron-event downloaderConfigure the event file to run the program nightly (file downloader as an example):bash-4.1$ cat ~/.hcron/hcron1.science.gc.ca/events/downloaderas_user=host=142.135.29.204command=bash -l -c /space/project/grdi/eco/groups/mbb/chz001/dataDownloader/hcron_command.shnotify_email=chunfang.zheng@canada.canotify_message=message from hcronwhen_month=*when_day=*when_hour=2when_minute=0when_dow=*template_name=bash-4.1$Command file(hcron_command.sh)bash-4.1$ cat /space/project/grdi/eco/groups/mbb/chz001/dataDownloader/hcron_command.sh#!/bin/bashecho "start atdate" >> $HOME/check_stepcd /space/project/grdi/eco/groups/mbb/chz001/dataDownloadersource venv/bin/activatelims_downloader -c config.iniecho "end atdate" >> $HOME/check_stepGetting Your Environment Rightfromhttps://expl.info/display/HCRON/Getting+Your+Environment+Right"The brute force way is to run a shell as a login shell, which will provide an environment almost equivalent to an interactive session"command=bash -l -c ""note: from my test, the trick works for host=142.135.29.204, but not for host=gpsc-in.science.gc.caConnect to hcron serverbash-4.1$ ssh hcron1.science.gc.caLoad your hcron eventschz001@hcron1: hcron-reloadSQLite databaseThree tables are maintained in this database. Tables will be updated when the program is run.data_packages: to keep all the information about each sequence run
(run-name,....)data_files: to keep all the information about each sequence file,
include information scrapped from webpage, checksum(SHA256), original name and new name of the file, etc.program_action: to keep all the information of every time the application is run,
like failures, successes, urls scraped/attempted, timestamps, sequence runs downloaded.