Package: ffq
ffq! NCBI is depracating .SRA file links. This may result in an empty list with `--ncbi`.+ Have a cool use case for ffq? Submit a PR to the `Use cases` section and we'll feature it!Fetch metadata information from the following databases:GEO: Gene Expression Omnibus,SRA: Sequence Read Archive,EMBL-EBI: European Molecular BIology Laboratory’s European BIoinformatics Institute,DDBJ: DNA Data Bank of Japan,NIH Biosample:  Biological source materials used in experimental assays,ENCODE: The Encyclopedia of DNA Elements.ffqreceives an accession and returns the metadata for that accession as well as the metadata for all downstream accessions following the connections between GEO, SRA, EMBL-EBI, DDBJ, and Biosample. If you useffqin a publication, please thecite*:Gálvez-Merchán, Á., et al. (2022). Metadata retrieval from sequence databases with ffq. bioRxiv 2022.05.18.492548.The manuscript is available here:https://doi.org/10.1101/2022.05.18.492548.By default, ffq returns all downstream metadata down to the level of the SRR record. However, the desired level of resolution can be specified.ffqcan also skip returning the metadata, and instead return the raw data download links from any available host (FTP,AWS,GCPorNCBI) for GEO and SRA ids.InstallationThe latest release can be installed withpipinstallffqThe development version can be installed withpipinstallgit+https://github.com/pachterlab/ffqUsageFetch information of an accession and display it in the terminalffq [accession]where[accession]is either:an SRA/EBI/DDJ accession(SRR,SRX,SRSorSRP)(ERR,ERX,ERSorERP)(DRR,DRS,DRXorDRP)a GEO accession (GSEorGSM)an ENCODE accession (ENCSR,ENCSBorENCSD)a Bioproject accession (CXR)a Biosample accession (SAMN)a DOIExamples:$ffqSRR9990627#=> Returns metadata for the SRR9990627 run.$ffqSRX7347523#=> Returns metadata for the experiment SRX7347523 and for its associated SRR run.$ffqGSE129845#=> Returns metadata for GSE129845 and for its 5 associated GSM, SRS, SRX and SRR ids.$ffqDRP004583#=> Returns metadata for the study DRP004583 and its 104 associated DRS, DRX and SRR ids.$ffqENCSR998WNE#=> Returns metadata for the ENCODE experiment ENCSR998WNE.Fetch information of multiple accessions and display it in the terminalffq [accession 1] [accession 2] ...where[accession 1]and[accession 2]are accessions belonging to any of the above usage example categories.Examples:$ffqSRR11181954SRR11181954SRR11181956#=> Returns metadata for the three SRR runs.$ffqGSM4339769GSM4339770GSM4339771#=> Returns metadata for the three GSM accessions, as well as for their corresponding downstream SRS, SRX and SRR accessions.Fetch information of an accession only down to specified levelffq -l [level] [accession]where[level]is the number of downstream accessions you want to fetchExamples:$ffq-l1GSM4339769#=> Returns metadata only for GSM4339769, and not from any downstream accession.$ffq-l3GSE115469#=> Returns metadata for GSE115469 and its downstream GSM and SRS accessions.Fetch only raw data links from the host of your choice and display it in the terminalFTP hostffq --ftp [accession(s)]where[accession(s)]is either a single accession or a space-delimited list of accessions.AWS hostffq --aws [accession(s)]GCP hostffq --gcp [accession(s)]NCBI hostffq --ncbi [accession(s)]Examples:# FTP with an SRR$ffq--ftpSRR10668798[{"accession":"SRR10668798","filename":"SRR10668798_1.fastq.gz","filetype":"fastq","filesize":31876537192,"filenumber":1,"md5":"bf8078b5a9cc62b0fee98059f5b87fa7","urltype":"ftp","url":"ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR106/098/SRR10668798/SRR10668798_1.fastq.gz"},
...# FTP with a GSE$ffq--ftpGSE115469[{"accession":"SRR7276474","filename":"P1TLH.bam","filetype":"bam","filesize":48545467653,"filenumber":1,"md5":"d0fde6bf21d9f97bdf349a3d6f0a8787","urltype":"ftp","url":"ftp://ftp.sra.ebi.ac.uk/vol1/SRA716/SRA716608/bam/P1TLH.bam"},
...# AWS with SRX$ffq--awsSRX7347523[{"accession":"SRR10668798","filename":"T84_S1_L001_R1_001.fastq.1","filetype":"fastq","filesize":null,"filenumber":1,"md5":null,"urltype":"aws","url":"s3://sra-pub-src-6/SRR10668798/T84_S1_L001_R1_001.fastq.1"},
...# GCP with ERS$ffq--gcpERS3861775[{"accession":"ERR3585496","filename":"4834STDY7002879.bam.1","filetype":"bam","filesize":null,"filenumber":1,"md5":null,"urltype":"gcp","url":"gs://sra-pub-src-17/ERR3585496/4834STDY7002879.bam.1"}]# NCBI with GSM$ffq--ncbiGSM2905292[{"accession":"SRR6425163","filename":"SRR6425163.1","filetype":"sra","filesize":null,"filenumber":1,"md5":null,"urltype":"ncbi","url":"https://sra-downloadb.be-md.ncbi.nlm.nih.gov/sos2/sra-pub-run-13/SRR6425163/SRR6425163.1"}]Write accession information to a single JSON fileffq -o [JSON_PATH] [accession(s)]where[JSON_PATH]is the path to the JSON file that will contain the information
and[accession(s)]is either a single accession or a space-delimited list of accessions.Write accession  information to multiple JSON files, one file per accessionffq -o [OUT_DIR] --split [accessions]where[OUT_DIR]is the path to directory to which to write the JSON files and[accessions]is a space-delimited list of accessions.
Information about each accession will be written to its own separate JSON file named[accession].json.Fetch information of all studies (and all of their runs) in one or more papersffq [DOIS]where[DOIS]is a space-delimited list of one or more DOIs. The output is a JSON-formatted string (or a JSON file if-ois provided) with SRA study accessions as keys. When--splitis also provided, each study is written to its own separate JSON.Complete output examplesExamples of complete outputs are available in theexamplesdirectory.Downloading dataffqis specifically designed to download metadata and to facilitate obtaining links to sequence files. To download raw data from the links obtained withffqyou can use one of the following:cURLandwgetfor FTP links,awsfor AWS links,gsutilfor GCP links,fasterq dumpfor converting SRA files to FASTQ files.FTPBy default,cURLis installed on most computers and can be used to download files with FTP links. Alternatively,wgetcan be used.# Obtain FTP links$ffq--ftpSRR10668798[{"accession":"SRR10668798","filename":"SRR10668798_1.fastq.gz","filetype":"fastq","filesize":31876537192,"filenumber":1,"md5":"bf8078b5a9cc62b0fee98059f5b87fa7","urltype":"ftp","url":"ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR106/098/SRR10668798/SRR10668798_1.fastq.gz"},{"accession":"SRR10668798","filename":"SRR10668798_2.fastq.gz","filetype":"fastq","filesize":43760586944,"filenumber":2,"md5":"351df47dca211c1f66ef327e280bd4fd","urltype":"ftp","url":"ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR106/098/SRR10668798/SRR10668798_2.fastq.gz"}]# Download the files one-by-one$curl-Oftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR106/098/SRR10668798/SRR10668798_1.fastq.gz$curl-Oftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR106/098/SRR10668798/SRR10668798_2.fastq.gzAlternatively, theurls can be extracted from the json output withjqand then piped intocURL.$ffq--ftpSRR10668798|jq-r'.[] | .url'|xargscurl-OIf you don't havejqinstalled, you can use the default programgrep.$ffq--ftpSRR10668798|grep-Eo'"url": "[^"]*"'|grep-o'"[^"]*"$'|xargscurl-OAWSIn order to download files from AWS, theawstool must be installed andcredentials must be setup.# Pipe AWS links to aws s3 cp and download$ffq--awsSRX7347523|jq-r'.[] | .url'|xargs-I{}awss3cp{}.GCPIn order to download files from GCP, thegsutiltool must be install andcredentials must be setup.# Pipe GCP links to gsutil cp and download$ffq--gcpERS3861775|jq-r'.[] | .url'|xargs-I{}gsutilcp{}.NCBI-SRASRA files downloaded from NCBI can be converted to FASTQ files usingfastq-dumpor the improvedfasterq-dumpboth of which are installed as part ofSRA Toolkit.# Pipe SRA link to curl and download the SRA file$ffq--ncbiGSM2905292|jq-r'.[] | .url'|xargscurl-O# Convert the SRA file to FASTQ files with one of the following$fastq-dump./SRR6425163.1--split-files--include-technical-O./SRR6425163--gzip$fasterq-dump./SRR6425163.1--split-files--include-technical-O./SRR6425163# fasterq-dump does not have gzip optionUse casesffqfacilitates the acquisition of publicly available sequencing data to help answer relevant research questions.# Goal: quantify publicly available scRNAseq data$pipinstallkb-pythonggetffq
$kbref-iindex.idx-gt2g.txt-f1transcriptome.fa$(ggetref--ftp-wdna,gtfhomo_sapiens)$kbcount-iindex.idx-gt2g.txt-x10xv3-oout$(ffq--ftpSRR10668798|jq-r'.[] | .url'|tr'\n'' ')# -> count matrix in out/ folder# Goal: count the total number of reads$ffqSRR10668798|jq'.. | ."ENA-SPOT-COUNT"? | select(. != null)'|paste-sd+-|bc624886427# Goal: check the total size of the FASTQ files$ffq--ftpSRR10668798|jq'.[] | .filesize 'blah|paste-sd+-|bc|numfmt--to=iec-i--suffix=B
71GiB# Goal: count the number of FASTQ files$ffq--ftpSRR10668798|jq-r'length'2# Goal: get sequence stats for the first 100 entries with seqkit$curl-s$(ffq--ftpSRR10668798|jq-r'.[0] | .url')|zcat|head-400|seqkitstats-a
fileformattypenum_seqssum_lenmin_lenavg_lenmax_lenQ1Q2Q3sum_gapN50Q20(%)Q30(%)-FASTQDNA1002,60026262613261302695.3192.92Submitted by@sbooeshaghi.# Goal: print the first 3 sequences of read 1 to the screen$curl-s$(ffq--ftpSRR10668798|jq-r'.[0] | .url')|zcat|awk'(NR-2)%4==0'|head-n
NCCAAATAGGAATTACATACACCCCC
NAACCTGAGTAGATGTGTTGTTAACT
NGATCTGAGAACTCGGAACTATTTTC# Goal: get number of counts per unique read sequence from the first 10000 reads$curl-s$(ffq--ftpaccession|jq-r'.[0] | .url')|zcat|awk'(NR-2)%4==0'|head-n10000|sort|uniq-c|sort-r4TACACGACACTTAACGATCGGCCTTC4GTACTTTAGGCCCGTTTGTGTGCGAT4GACGGCTAGTACATGATATAACAAGC
...Submitted by@agalvezm.Do you have a cool use case forffq? Submit a PR (including the goal, code snippet, and your username) so that we can feature it here.Failure modesMany factors, independent offfq, may result in failure to fetch metadata or missing metadata including:broken internet connectionimproperly formatted accessionrecently submitted data to SRA (not synced with ENA)exceeded request rate for serversmissing metadata from online databaseIf you believe you have identified a bug inffqplease see the section oncontributing*.ContributingThank you for wanting to improveffq! If you have a bug that is related toffqplease create an issue. The issue should containtheffqcommand ran with--verbose,the error message, andtheffqandpythonversion.Please make all Pull Requests against thedevelbranch and include a message detailing the exact changes made, the reasons for the change, and tests that check for the correctness of those changes.Some tips for improving theffqcode base:the developer dependencies can be installed withpip install -r dev-requirements.txtunit tests can be added to the./tests/test_*.pycode reformatting can be performed by runningblack ffq/code quality can be checked by runningmake checktests can be performed by runningmake testCaveats and limitationsffqrelies on the information provided by the different APIs it uses to retrieve metadata (hosted by ENA, NCBI, ENCODE, etc). Therefore, returning consistent and accurate metadata is dependent on the accuracy and consistency of such databases. Unfortunately, we have observed instances where some APIs are updated without notice. This leads to unconsistent metadata retrieval by ffq that cannot be solved on our end.For example, as of May 29th, the command:ffq--ncbiSRR6835844returned:[{'accession':'SRR6835844','filename':'SRR6835844.1','filenumber':1,'filesize':None,'filetype':'sra','md5':None,'url':'https://sra-downloadb.be-md.ncbi.nlm.nih.gov/sos2/sra-pub-run-13/SRR6835844/SRR6835844.1','urltype':'ncbi'}]On June 1st, we detected an error in one of ffq’s tests. Running the same command led to the following output:[]Investigating this issue, we discovered that the output of the eutil’s efetch tool had changed (for a comparison, compare filesSRR6835844_altlinks_old.txtandSRR6835844_altlinks_new.txtcontained intests/fixtures). In the new output, ncbi hosted links were no longer provided. This affects a large number of accessions, not only SRR6835844. We have updated our tests accordingly and will continue to monitor the situation.Namingffqis short for FetchFastQ.Cite@article{galvez2022metadata,
  title={Metadata retrieval from sequence databases with ffq},
  author={G{\'a}lvez-Merch{\'a}n, {\'A}ngel and Min, Kyung Hoi Joseph and Pachter, Lior and Booeshaghi, A. Sina},
  year={2022}
}