Package: bdss-client
The increasing size of datasets used in scientific computing has made it difficult or impossible for a
researcher to store all their data at the compute site they are using to process it. This has necessitated
that a data transfer step become a key consideration in experimental design. Accordingly, scientific data
repositories such as NCBI have begun to offer services such as dedicated data transfer machines and advanced
transfer clients. Despite this, many researchers continue familiar but suboptimal practices: using slow
transfer clients like a web browser orscp, transferring data over wireless networks, etc.BDSS aims to alleviate this problem by shifting the burden of learning about alternative file mirrors,
transfer clients, tuning parameters, etc. from the end user researcher to a group of “data curators”. It
consists of three parts:ComponentsMetadata repositoryCentral database managed by data curatorsMatches patterns of data file URLs and maps them to alternate sourcesIncludes information about the transfer tool to use to retrieve the dataBDSS transfer clientConsumes information from metadata repositoryInvokes transfer toolsReports analytics to metadata repositoryIntegration as aGalaxydata transfer toolGet StartedMoving data with the BDSS client:From the command line(Video)From Galaxy(Video)Managing an existing metadata repositorySetting up a new metadata repositoryFor development or testingFor production useExamplesAll examples here require a metadata repository configured to support them. The default metadata repository
athttps://bdss.bioinfo.wsu.edu/supports these examples and the necessary configuration is also listed here.NCBI SRA archiveJGI Genome PortalNCBI SRA archiveNCBI makes files available fortransfer using Aspera Connect,
a tool with “improved data transfer characteristics” vs FTP or HTTP. Ifascpis installed on your machine,
BDSS can handle building the appropriate command.Without BDSS:ascp-i$HOME/.aspera/connect/etc/asperaweb_id_dsa.openssh-Tanonftp@ftp.ncbi.nlm.nih.gov:/sra/sra-instant/reads/ByRun/sra/SRR/SRR039/SRR039885/SRR039885.sra./With BDSS:bdsstransfer-u'ftp://ftp.ncbi.nlm.nih.gov/sra/sra-instant/reads/ByRun/sra/SRR/SRR039/SRR039885/SRR039885.sra'Metadata repository configuration:{"data_sources":[{"description":"","label":"NCBI Sequence Read Archive with FTP","test_files":[],"transfer_mechanism":{"options":{},"type":"curl"},"transforms":[{"for_destinations":[],"options":{"new_scheme":"aspera"},"target":"NCBI Sequence Read Archive with Aspera","type":"change_scheme"}],"url_matchers":[{"options":{"pattern":"^ftp://ftp\\.ncbi\\.nlm\\.nih\\.gov/sra"},"type":"regular_expression"}]},{"description":"","label":"NCBI Sequence Read Archive with Aspera","test_files":[],"transfer_mechanism":{"options":{"disable_encryption":true,"username":"anonftp"},"type":"aspera"},"transforms":[],"url_matchers":[{"options":{"pattern":"^aspera://ftp\\.ncbi\\.nlm\\.nih\\.gov/sra"},"type":"regular_expression"}]}],"destinations":[]}JGI Genome PortalTo download files from theJGI Genome Portal, you must firstauthenticate. BDSS can prompt for credentials and handle
storing your session cookies.Without BDSS:curl'https://signon.jgi.doe.gov/signon/create'--data-urlencode'login=USER_NAME'--data-urlencode'password=USER_PASSWORD'-ccookies>/dev/nullcurl'http://genome.jgi.doe.gov/ext-api/downloads/get-directory?organism=PhytozomeV10'-bcookies>get-directoryWith BDSS:bdsstransfer-u'http://genome.jgi.doe.gov/ext-api/downloads/get-directory?organism=PhytozomeV10'JGIGenomePortalusername?USER_NAMEJGIGenomePortalpassword?USER_PASSWORDMetadata repository configuration:{"data_sources":[{"description":"","label":"JGI Genome Portal","test_files":[],"transfer_mechanism":{"options":{"auth_url":"https://signon.jgi.doe.gov/signon/create","password_field":"password","password_prompt":"JGI Genome Portal password?","username_field":"login","username_prompt":"JGI Genome Portal username?"},"type":"session_authenticated_curl"},"transforms":[],"url_matchers":[{"options":{"pattern":"http:\\/\\/genome\\.jgi\\.doe\\.gov\\/ext-api"},"type":"regular_expression"}]}],"destinations":[]}