Package: fastq-downloader
Fastq Downloader (WIP)This python package let you download fastq files from ena.
It can automatic merge and rename fastq files based on the input file provided.
If you have trouble downloading this repo's release, please go tofastgitto see if this project can help you.Notice for ReadmeIf you are reading this from pypi, please go togithubto read the latest readme file, for I won't modify pypi readme unless new version released.How to UseInstallationcondacreate--namefastq-downloader-cconda-forge-chcc-cbiocondaaspera-clisnakemake-minimalhttpxlxmlclickbeautifulsoup4python=3.9## use what ever you want to download the gist mentioned above to thisname.smkcondaactivatefastq-downloader
pipinstallfastq-downloader==0.4.4Usagemake sure to create aninfo.tsvbefore, you can just copy from theGEOwebsite,
then go to vim, type:setpaste to get into paste mode,
paste the table into vim,
and you can modify the names of samples to suit your need,
the downloaded file will then be renamed too.
Save the file as whatever name you want, then exit vim
the white space will be auto convert to underscorefirst, we have to turn info tsv to individual sample accession files## step 1## you can use fastq-downloader breakdown-tsv --help to view the helpfastq-downloaderbreakdown-tsv\-ipath/to/info.tsv\-opath/to/output/dirAll paths can be relative paths.Then we can start to download.## step 2fastq-downloadersmk\-ipath/to/info.tsv\-opath/to/output/dir\-t{number_of_threads_you_want}\--download-backendascpafter the download is done, you can usefindcommand to get all of thefastq.gzfiles and link them to anoter place. For example I have a bunch of file downloaded todownloadfolder, the folder structure should look like this:# this is what inside download folder
. 
└── merged_fastq
    ├── GSM5159835
    │   ├── wt_1_R1.fastq.gz
    │   └── wt_1_R2.fastq.gz
    ├── GSM5159836
    │   ├── wt_2_R1.fastq.gz
    │   └── wt_2_R2.fastq.gz
    └── GSM5159837
        ├── wt_3_R1.fastq.gz
        └── wt_3_R2.fastq.gzThen executefind -name "*fastq.gz" | xargs -I {} ln -s {} .Allfastq.gzfiles will be linked to the root ofdownloadfolder:.
├── merged_fastq
│   ├── GSM5159835
│   │   ├── wt_1_R1.fastq.gz
│   │   └── wt_1_R2.fastq.gz
│   ├── GSM5159836
│   │   ├── wt_2_R1.fastq.gz
│   │   └── wt_2_R2.fastq.gz
│   └── GSM5159837
│       ├── wt_3_R1.fastq.gz
│       └── wt_3_R2.fastq.gz
├── wt_1_R1.fastq.gz -> merged_fastq/GSM5159835/wt_1_R1.fastq.gz
├── wt_1_R2.fastq.gz -> merged_fastq/GSM5159835/wt_1_R2.fastq.gz
├── wt_2_R1.fastq.gz -> merged_fastq/GSM5159836/wt_2_R1.fastq.gz
├── wt_2_R2.fastq.gz -> merged_fastq/GSM5159836/wt_2_R2.fastq.gz
├── wt_3_R1.fastq.gz -> merged_fastq/GSM5159837/wt_3_R1.fastq.gz
└── wt_3_R2.fastq.gz -> merged_fastq/GSM5159837/wt_3_R2.fastq.gzThis should add some convinience for your subsequent process.These command lines should suit your need in most situations,
for those who want more flexiblity and control to the underlyingsnakemakeworkflow,
you can append your argument to the-soption of thesmksubcommand;
or you can directly use the snakemake file in this repo.For other advanced use you can always use--help, or read the source code.It will automatically try to download the file, check md5, retry if file integrity check failed, and merge the files if the number of files is more than 2, finally rename the files to the description you provided.prepare the info.tsv like this:
note the file must be tab delimited (tsv file), you can simply achieve this by paste it from the Excel or GEO website. Or from SRA Run Selector downloaded csv file.GSM12345  h3k9me3_rep1
GSM12345  h3k9me3_rep2Notice for Commonly Encountered Problemserror fromascpsayingfailed to authenticate:It can be a network issue according tothis issue on githubor a server issue of EBIthis post on biostarIf you have encountered this problem, please try to delete the download target folder and change the--download-backendargument towgetto use ftp links.Todotest for paired-end reads run mergepublish to biocondaif fail, retryuse dag to run the pipeline (sort of, implemented by using snakemake)option to resume download when md5 not matchoption to continue from last time downloadimplement second level parallelizationKnown IssuesWill fail to download the files contains both paired-end reads and single-end reads. (yes it exists).Update Content0.4.4:Bump version to trigger pypi readme updateFix version number.0.4.3:Update readme.Breakdown the download process to two steps and add new download backend andwget.0.3.2:Add filter for library layout (some sra entry has content mismatches its library layout)