Package: ncbi-genome-download
NCBI Genome Downloading ScriptsSome script to download bacterial and fungal genomes from NCBI after they
restructured their FTP a while ago.Idea shamelessly stolen fromMick Watson's Kraken downloader
scriptsthat can also be found inMick's GitHub
repo. However, Mick's
scripts arewritten in Perlspecific to actually building a Kraken database
(as advertised).So this is a set of scripts that focuses on the actual genome downloading.Installationpipinstallncbi-genome-downloadAlternatively, clone this repository from GitHub, then run (in a python virtual environment)pipinstall.If this fails on older versions of Python, try updating yourpiptool first:pipinstall--upgradepipand then rerun thencbi-genome-downloadinstall.Alternatively,ncbi-genome-downloadis packaged inconda.
Refer the the Anaconda/minicondasite to
install a distribution (highly recommended). With that installed one can do:condainstall-cbiocondancbi-genome-downloadncbi-genome-downloadis only developed and tested on Python releases still
under active support by the Python project. At the moment, this means versions
3.7, 3.8, 3.9, 3.10 and 3.11.
Specifically, no attempt at testing under Python versions older than 3.7 is
being made.If your system is stuck on an older version of Python, consider using a tool likeHomebrewto obtain a more up-to-date version.ncbi-genome-download0.2.12 was the last version to support Python 2.UsageTo download all bacterial RefSeq genomes in GenBank format from NCBI, run the following:ncbi-genome-downloadbacteriaDownloading multiple groups is also possible:ncbi-genome-downloadbacteria,viralNote: To see all available groups, seencbi-genome-download --help, or
simply useallto check all groups. Naming a more specific group will reduce
the download size and the time needed to find the sequences to download.If you're on a reasonably fast connection, you might want to try running
multiple downloads in parallel:ncbi-genome-downloadbacteria--parallel4To download all fungal GenBank genomes from NCBI in GenBank format, run:ncbi-genome-download--sectiongenbankfungiTo download all viral RefSeq genomes in FASTA format, run:ncbi-genome-download--formatsfastaviralIt is possible to download multiple formats by supplying a list of formats or
simply downloading all formats:ncbi-genome-download--formatsfasta,assembly-reportviral
ncbi-genome-download--formatsallviralTo download only completed bacterial RefSeq genomes in GenBank format, run:ncbi-genome-download--assembly-levelscompletebacteriaIt is possible to download multiple assembly levels at once by supplying a list:ncbi-genome-download--assembly-levelscomplete,chromosomebacteriaTo download only bacterial reference genomes from RefSeq in GenBank format, run:ncbi-genome-download--refseq-categoriesreferencebacteriaTo download bacterial RefSeq genomes of the genusStreptomyces, run:ncbi-genome-download--generaStreptomycesbacteriaNote: This is a simple string match on the organism name provided by NCBI only.You can also use this with a slight trick to download genomes of a certain
species as well:ncbi-genome-download--genera"Streptomyces coelicolor"bacteriaNote: The quotes are important. Again, this is a simple string match on the organism
name provided by the NCBI.Multiple genera is also possible:ncbi-genome-download--genera"Streptomyces coelicolor,Escherichia coli"bacteriaYou can also put genus names into a file, one organism per line, e.g.:Streptomyces
AmycolatopsisThen, pass the path to that file (e.g.my_genera.txt) to the--generaoption, like so:ncbi-genome-download--generamy_genera.txtbacteriaNote: The above command will download allStreptomycesandAmycolatopsisgenomes from RefSeq.You can make the string match fuzzy using the--fuzzy-genusoption. This can
be handy if you need to match a value in the middle of the NCBI organism name,
like so:ncbi-genome-download--generacoelicolor--fuzzy-genusbacteriaNote: The above command will download all bacterial genomes containing
"coelicolor" anywhere in their organism name from RefSeq.To download bacterial RefSeq genomes based on their NCBI species taxonomy ID, run:ncbi-genome-download--species-taxids562bacteriaNote: The above command will download all RefSeq genomes belonging toEscherichia coli.To download a specific bacterial RefSeq genomes based on its NCBI taxonomy ID, run:ncbi-genome-download--taxids511145bacteriaNote: The above command will download the RefSeq genome belonging toEscherichia coli str. K-12 substr. MG1655.It is also possible to download multiple species taxids or taxids by supplying
the numbers in a comma-separated list:ncbi-genome-download--taxids9606,9685--assembly-levelchromosomevertebrate_mammalianNote: The above command will download the reference genomes for cat and human.In addition, you can put multiple species taxids or taxids into a file, one per line
and pass that filename to the--species-taxidsor--taxidsparameters, respectively.Assuming you had a filemy_taxids.txtwith the following contents:9606
9685You could download the reference genomes for cat and human like this:ncbi-genome-download--taxidsmy_taxids.txt--assembly-levelschromosomevertebrate_mammalianIt is possible to also create a human-readable directory structure in parallel
to mirroring the layout used by NCBI:ncbi-genome-download--human-readablebacteriaThis will use links to point to the appropriate files in the NCBI directory structure,
so it saves file space. Note that links are not supported on some Windows file
systems and some older versions of Windows.It is also possible to re-run a previous download with the--human-readableoption.
In this case,ncbi-genome-downloadwill not download any new genome files, and
just create human-readable directory structure. Note that if any files have been
changed on the NCBI side, a file download will be triggered.There is a "dry-run" option to show which accessions would be downloaded, given
your filters:ncbi-genome-download--dry-runbacteriaIf you want to filter for the "relation to type material" column of the
assembly summary file, you can use the--type-materialsoption. Possible
values are "any", "all", "type", "reference", "synonym", "proxytype", and/or
"neotype". "any" will include assemblies with no relation to type material
value defined, "all" will download only assemblies with a defined value.
Multiple values can be given, separated by comma:ncbi-genome-download--type-materialstype,referenceBy default, ncbi-genome-download caches the assembly summary files for the
respective taxonomic groups for one day. You can skip using the cache file by
using the--no-cacheoption. The output of--helpalso shows the cache
directory, should you want to remove any of the cached files.To get an overview of all options, runncbi-genome-download--helpAs a methodYou can also use it as a method call. Pass the pythonised keyword arguments
(_instead of-) as described above or in the--help:importncbi_genome_downloadasngdngd.download()Note: To specify a taxonomic group, likebacteria, use thegroupkeyword.Contributed Scripts:gimme_taxa.pyThis script lets you find out what TaxIDs to pass tongd, and will write a
simple one-item-per-line file to pass in to it. It utilises theete3toolkit,
so refer to their site to install the dependency if it's not already satisfied.You can query the database using a particular TaxID, or a scientific name. The
primary function of the script is to return all the child taxa of the specified
parent taxa. The script has various options for what information is written in
the output.A basic invocation may look like:# Fetch all descendent taxa for Escherichia (taxid 561):pythongimme_taxa.py-o~/mytaxafile.txt561# Alternatively, just provide the taxon namepythongimme_taxa.py-oall_descendent_taxids.txtEscherichia# You can provide multiple taxids and/or namespythongimme_taxa.py-oall_descendent_taxids.txt561,MethanobrevibacterOn first use, a small sqlite database will be created in your home directory
by default (change the location with the--databaseflag). You can update this
database by using the--updateflag. Note that if the database is not in your
home directory, you must specify it with--databaseor a new database will be
created in your home directory.To see all help:pythongimme_taxa.py
pythongimme_taxa.py-h
pythongimme_taxa.py--helpCitingncbi-genome-downloadYou can citencbi-genome-downloadvia the Zenodo deposit underDOI: 10.5281/zenodo.8192433.LicenseAll code is available under the Apache License version 2, see theLICENSEfile for details.