Package: pyega3
EGA download client: pyEGA3OverviewThe pyEGA3 download client is a python-based tool for viewing and downloading files from authorized EGA datasets. pyEGA3 uses the EGA Data API and has several key features:Files are transferred over secure https connections and received unencrypted, so no need for decryption after download.Downloads resume from where they left off in the event that the connection is interrupted.pyEGA3 supports file segmenting and parallelized download of segments, improving overall performance.After download completes, file integrity is verified using checksums.pyEGA3 implements the GA4GH-compliant htsget protocol for download of genomic ranges for data files with accompanying index files.Tutorial videoA video tutorial demonstrating the usage of pyEGA3 from installation through file download is availablehere.RequirementsPython 3.6 or newer. (download instructions)Installation and updateUsing Pip3Install pyEGA3 using pip3.sudopip3installpyega3Update pyEGA3, if needed, using pip3.pip3installpyega3--upgradeTest your pip3 installation by running pyEGA3.pyega3--helpUsing conda (bioconda channel)Install pyEGA3 using conda.condaconfig--addchannelsbioconda
condaconfig--addchannelsconda-forge
condainstallpyega3Update pyEGA3, if needed, using conda.condaupdatepyega3Test your conda installation by running pyEGA3.pyega3--helpUsing GitHubClone theega-download-clientGitHub repository.Navigate to the directory where the repository was cloned.cdpath/to/ega-download-clientThree scripts are provided to install the required Python environment depending on the host operating system.Linux (Red Hat): red_hat_dependency_install.shLinux: debian_dependency_install.shmacOS: osx_dependency_install.shExecute the script corresponding to the host operating system. For example, if using Red Hat Linux, run:shred_hat_dependency_install.shTest your GitHub installation by running pyEGA3.python-mpyega3.pyega3--helpUsing DockerThere are Docker images built by Bioconda:https://bioconda.github.io/recipes/pyega3/README.htmlAn example of running pyEGA3 in a Docker container:dockerrun--rm-v/tmp:/app-w/appquay.io/biocontainers/pyega3:3.4.0--py_0pyega3-d-tfetchEGAF00001775036This example command mounts your /tmp folder into the Docker container as /app,
starts the 3.4.0 version of pyEGA3 and downloads a test file.
The test file will be downloaded into your /tmp folder.
You can find other, possibly newer, versions ("tags") of the pyEGA3 Docker image
on the above-mentioned Bioconda page.Usage - File downloadusage:pyega3.py[-h][-d][-cfCONFIG_FILE][-sfSERVER_FILE][-cCONNECTIONS][-t][-msMAX_SLICE_SIZE]{datasets,files,fetch}...

DownloadfromEMBLEBI'sEGA(EuropeanGenome-phenomeArchive)positionalarguments:{datasets,files,fetch}subcommandsdatasetsListauthorizeddatasetsfilesListfilesinaspecifieddatasetfetchFetchadatasetorfile

optionalarguments:-h,--helpshowthishelpmessageandexit-d,--debugExtradebuggingmessages-cfCONFIG_FILE,--config-fileCONFIG_FILEJSONfilecontainingcredentials/confige.g.{"username":"user1","password":"toor"}-sfSERVER_FILE,--server-fileSERVER_FILEJSONfilecontainingserverconfige.g.{"url_auth":"aai url","url_api":"api url","url_api_ticket":"htsget url","client_secret":"client secret"}-cCONNECTIONS,--connectionsCONNECTIONSDownloadusingspecifiednumberofconnections(default:1connection)-t,--testTestuseractivated-msMAX_SLICE_SIZE,--max-slice-sizeMAX_SLICE_SIZESetmaximumsizeforeachsliceinbytes(default:100MB)Testing pyEGA3 installationWe recommend that all fresh installations of pyEGA3 be tested. A test account has been created which can be used (-t) to test the following pyEGA3 actions:List the datasets available to the test accountpyega3-d-tdatasetsList the files available in a test datasetpyega3-d-tfilesEGAD00001003338Download a test filepyega3-d-tfetchEGAF00001775036The test dataset (EGAD00001003338) is large (almost 1TB), so please be mindful if deciding to test downloading the entire dataset. The test account does not require an EGA username and password because it contains publicaly accessible files from the1000 Genomes Project. The files in the test dataset can be used for troubleshooting and training purposes.Defining credentialsTo view and download files for which you have been granted access, pyEGA3 requires your EGA username (email address) and password saved to a credentials file.Create a file called CREDENTIALS_FILE and place it in the directory where pyEGA3 will run. The credentials file must be in JSON format and must contain your registered EGA username (email address) and password provided by EGA Helpdesk.An example CREDENTIALS_FILE is availablehere.Using pyEGA3 for file downloadReplace<these values>with values relevant for your datasets.Display authorized datasetspyega3-cf</Path/To/CREDENTIALS_FILE>datasetsDisplay files in a datasetpyega3-cf</Path/To/CREDENTIALS_FILE>filesEGAD<NUM>Download a datasetpyega3-cf</Path/To/CREDENTIALS_FILE>fetchEGAD<NUM>--output-dir</Path/To/OutputDirectory>Download a single filepyega3-cf</Path/To/CREDENTIALS_FILE>fetchEGAF<NUM>--output-dir</Path/To/OutputDirectory>List unencrypted md5 checksums for all files in a datasetpyega3-cf</Path/To/CREDENTIALS_FILE>filesEGAD<NUM>Save unencrypted md5 checksums to a filenohuppyega3-cf</Path/To/CREDENTIALS_FILE>filesEGAD<NUM></Path/To/File/md5sums.txt>Download a file or dataset using 5 connectionspyega3-c5-cf</Path/To/CREDENTIALS_FILE>fetchEGAD<NUM>--output-dir</Path/To/OutputDirectory>Usage - Genomic range requests via htsgetusage:pyega3fetch[-h][--reference-nameREFERENCE_NAME][--reference-md5REFERENCE_MD5][--startSTART][--endEND][--format{BAM,CRAM,VCF,BCF}][--max-retriesMAX_RETRIES][--retry-waitRETRY_WAIT][--output-dirOUTPUT_DIR][--delete-temp-files]identifier

positionalarguments:identifierIdfordataset(e.g.EGAD00000000001)orfile(e.g.EGAF12345678901)optionalarguments:-h,--helpshowthishelpmessageandexit--reference-nameREFERENCE_NAME,-rREFERENCE_NAMEThereferencesequencename,forexample'chr1','1',or'chrX'.Ifunspecified,alldataisreturned.--reference-md5REFERENCE_MD5,-mREFERENCE_MD5TheMD5checksumuniquelyrepresentingtherequestedreferencesequenceasalower-casehexadecimalstring,calculatedastheMD5oftheupper-casesequenceexcludingallwhitespacecharacters.--startSTART,-sSTARTThestartpositionoftherangeonthereference,0-based,inclusive.Ifspecified,reference-nameorreference-md5mustalsobespecified.--endEND,-eENDTheendpositionoftherangeonthereference,0-basedexclusive.Ifspecified,reference-nameorreference-md5mustalsobespecified.--format{BAM,CRAM,VCF,BCF},-f{BAM,CRAM,VCF,BCF}Theformatofdatatorequest.--max-retriesMAX_RETRIES,-MMAX_RETRIESThemaximumnumberoftimestoretryafailedtransfer.Anynegativenumbermeansinfinitenumberofretries.--retry-waitRETRY_WAIT,-WRETRY_WAITThenumberofsecondstowaitbeforeretryingafailedtransfer.--output-dirOUTPUT_DIROutputdirectory.Thefileswillbesavedintothisdirectory.Mustexist.Default:thecurrentworkingdirectory.--delete-temp-filesDonotkeepthosetemporary,partialfileswhichwereleftonthediskafterafailedtransfer.Using pyEGA3 for fetching a genomic rangeReplace<these values>with values relevant for your datasets. Please note that htsget can only be used with files that have corresponding index files in EGA.Download chromosome 1 for a BAM filepyega3fetch-cf</Path/To/CREDENTIALS_FILE>--reference-name1--formatBAM--output-dir</Path/To/OutputDirectory>EGAF<NUM>Download position 0-1000000 on chromosome 1 for a BAM filepyega3fetch-cf</Path/To/CREDENTIALS_FILE>--start0--end1000000--reference-name1--formatBAM--output-dir</Path/To/OutputDirectory>EGAF<NUM>TroubleshootingFirst, please ensure you are using the most recent version of pyEGA3 by following instructions in the "Installation and update" section for updating pyEGA3.Failure to validate credentialsPlease ensure that your credentials are formatted correctly. Email addresses (usernames) are case-sensitive. If you have an EGA submission account, these credentials are different from your data access credentials. Please ensure you are using your data access credentials with pyEGA3.Slow download speedsDownload speed can be optimized using the--connectionsparameter which will parallelize download at the file level. If the--connectionsparameter is provided, all files >100Mb will be downloaded using the specified number of parallel connections.Using a very high number of connections will introduce overhead that can slow the download of the file. It is important to note that files are still downloaded sequentially, so using multiple connections does not mean downloading multiple files in parallel. We recommend trying with 30 connections initially and adjusting from there to get maximum throughput.File taking a long time to savePlease note that when a file is being saved, it goes through two processes. First, the downloaded file "chunks" are pieced back together to reconstruct the original file. Second, pyEGA3 calculates the checksum of the file to confirm the file downloaded successfully. Larger files will take more time to reconstruct and validate the checksum.--saveto argument is not recognisedThe--savetocommand-line argument is now called--output-dirand, in contrast with the original--savetoargument, one can only specify now an output directory, but not an output file.This change was made to improve the user experience and avoid issues caused by supplying a filename
where a directory was expected, and vice versa.Thus, the original command, where the--savetoargument specified a directory:pyega3-cf</Path/To/CREDENTIALS_FILE>fetchEGAF<NUM>--saveto</Path/To/OutputDirectory>should be rewritten like this:pyega3-cf</Path/To/CREDENTIALS_FILE>fetchEGAF<NUM>--output-dir</Path/To/OutputDirectory>The original command:pyega3-cf</Path/To/CREDENTIALS_FILE>fetchEGAF<NUM>--saveto</Path/To/NewFileName>where the--savetoargument specified a new file-name, is no longer supported,
because the--output-dirargument supports only output directories, but not output files.Further assistanceIf, after troubleshooting an issue, you are still experiencing difficulties, please email EGA Helpdesk (helpdesk@ega-archive.org) with the following information:Attach the log file (pyega3_output.log) located in the directory where pyEGA3 is runningIndicate the compute environment you are running pyEGA3 in: compute cluster, single machine, other (please describe).AttributionParts of pyEGA3 are derived frompyEGAdeveloped by James Blachly.