Package: genomepy
genomepy: genes and genomes at your fingertipsgenomepy is designed to provide asimpleandstraightforwardway to download and use genomic data.
This includes (1) searching available data,
(2) showing the available metadata,
(3) automatically downloading, preprocessing and matching data and
(4) generating optional aligner indexes.
All with sensible, yet controllable defaults.
Currently, genomepy supports Ensembl, UCSC, NCBI and GENCODE.Pssst, hey there!Is genomepy not doing what you want? Does it fail? Is it clunky?
Is the documentation unclear? Have any other ideas on how to improve it?
Don't be shy andlet us know!Table of ContentsInstallationQuick usageCommand lineSearchInspectInstallPluginsPythonFrequently Asked QuestionsGetting helpContributingCitationLicenseInstallationgenomepy requires Python 3.7+You can install genomepy viabioconda, pip or git.Bioconda$condainstall-cconda-forge-cbioconda'genomepy>=0.15'Pip$pipinstallgenomepyWith the Pip installation, you will have to install additional dependencies, and make them available in your PATH.To read/write bgzipped genomes you will have to installpysam.If you want to use gene annotation features, you will have to install the following utilities:genePredToBedgenePredToGtfbedToGenePredgtfToGenePredgff3ToGenePredYou can find the binarieshere.Git$gitclonehttps://github.com/vanheeringen-lab/genomepy.git
$condaenvcreate-ngenomepy-fgenomepy/environment.yml
$condaactivategenomepy
$pipinstall-egenomepyQuick usageFind your genome:$ genomepy search zebrafishConsole output:nameprovideraccessiontax_idannotationspeciesother_info
GRCz11EnsemblGCA_000002035.47955✓Daniorerio2017-08-Ensembl/2018-04^UsenameforgenomepyinstallInstall your genome (with gene annotation):$ genomepy install --annotation GRCz11 --provider ensemblThe default genomes directory:~/.local/share/genomes/Command line interfaceAll commands come with a short explanation when appended with-h/--help.$genomepy--help
Usage:genomepy[OPTIONS]COMMAND[ARGS]...

Options:--versionShowtheversionandexit.-h,--helpShowthismessageandexit.

Commands:annotationshow1stlinesofeachannotationcleanremoveproviderdataconfigmanageconfigurationgenomeslistavailablegenomesinstallinstallagenome&runactivepluginspluginmanagepluginsproviderslistavailableproviderssearchsearchforgenomesSearch genomes & gene annotationsLet's say we want to download aXenopus tropicalisgenome & gene annotation.
First, lets find out what's out there!You can search by name, taxonomy ID or assembly accession ID.
Additionally, you can limit the search result to one provider with-p/--provider.
Furthermore, you can get the absolute--sizeof each genome (this option slows down the search).$genomepysearchxenopustro
nameprovideraccessiontax_idannotationspeciesother_infonrek
Xenopus_tropicalis_v9.1EnsemblGCA_000004195.38364✓Xenopustropicalis2019-04-Ensembl/2019-12
xenTro1UCSCna8364✗✗✗✗XenopustropicalisOct.2004(JGI3.0/xenTro1)xenTro2UCSCna8364✗✓✓✗XenopustropicalisAug.2005(JGI4.1/xenTro2)xenTro3UCSCGCA_000004195.18364✗✓✓✗XenopustropicalisNov.2009(JGI4.2/xenTro3)xenTro7UCSCGCA_000004195.28364✓✓✗✗XenopustropicalisSep.2012(JGI7.0/xenTro7)xenTro9UCSCGCA_000004195.38364✓✓✓✗XenopustropicalisJul.2016(Xenopus_tropicalis_v9.1/xenTro9)Xtropicalis_v7NCBIGCF_000004195.28364✓XenopustropicalisDOEJointGenomeInstitute
Xenopus_tropicalis_v9.1NCBIGCF_000004195.38364✓XenopustropicalisDOEJointGenomeInstitute
UCB_Xtro_10.0NCBIGCF_000004195.48364✓XenopustropicalisUniversityofCalifornia,Berkeley
ASM1336827v1NCBIGCA_013368275.18364✗XenopustropicalisSouthernUniversityofScienceandTechnology^UsenameforgenomepyinstallInspect gene annotationsLet's say we want to download theXenopus tropicalisgenome & gene annotation from UCSC.Since we are interested in the gene annotation as well, we should check which gene annotation suits our needs.
As you can see in the search results, UCSC has several gene annotations for us to choose from.
In the search results,n r e kdenotes which UCSC annotations are available.
These stand forncbiRefSeq,refGene,ensGene andknownGene, respectively.We can quickly inspect these with thegenomepy annotationcommand:$genomepyannotationxenTro9-pucsc12:04:41|INFO|UCSCncbiRefSeq
chr1genomepytranscript133270152620.-.gene_id"LOC100490505";transcript_id"XM_012956089.1";gene_name"LOC100490505";chr1genomepyexon133270134186.-.gene_id"LOC100490505";transcript_id"XM_012956089.1";exon_number"1";exon_id"XM_012956089.1.1";gene_name"LOC100490505";12:04:45|INFO|UCSCrefGene
chr1genomepytranscript193109390193134311.+.gene_id"pias2";transcript_id"NM_001078987";gene_name"pias2";chr1genomepyexon193109390193109458.+.gene_id"pias2";transcript_id"NM_001078987";exon_number"1";exon_id"NM_001078987.1";gene_name"pias2";12:04:49|INFO|UCSCensGene
chr1genomepytranscript133270152620.-.gene_id"ENSXETG00000030302.2";transcript_id"ENSXETT00000061673.2";gene_name"ENSXETG00000030302.2";chr1genomepyexon133270134186.-.gene_id"ENSXETG00000030302.2";transcript_id"ENSXETT00000061673.2";exon_number"1";exon_id"ENSXETT00000061673.2.1";gene_name"ENSXETG00000030302.2";Here we can see that therefGeneannotation has actual HGNC gene names, so lets go with this annotation.
This differs between assemblies, so be sure to check!Install a genome & gene annotationCopy the name returned by the search function to install.$genomepyinstallxenTro9You can choose to download gene annotation files with the-a/--annotationoption.$genomepyinstallxenTro9--annotationFor UCSC we can also select the annotation type.
Seegenomepy install --helpfor all provider specific options.$genomepyinstallxenTro9--UCSC-annotationrefGeneSince we did not specify the provider here, genomepy will use the first provider withxenTro9.
You can specify a provider by name with-p/--provider:$genomepyinstallxenTro9-pUCSC
Downloadinggenomefromhttp://hgdownload.soe.ucsc.edu/goldenPath/xenTro9/bigZips/xenTro9.fa.gz...
Genomedownloadsuccessful,startingpostprocessing...

name:xenTro9localname:xenTro9
fasta:~/.local/share/genomes/xenTro9/xenTro9.faNext, the genome is downloaded to the directory specified in the config file (by default~/.local/share/genomes).
To choose a different directory, use the-g/--genomes_diroption:$genomepyinstallsacCer3-pUCSC-g/path/to/my/genomes
Downloadinggenomefromhttp://hgdownload.soe.ucsc.edu/goldenPath/sacCer3/bigZips/chromFa.tar.gz...
Genomedownloadsuccessful,startingpostprocessing...

name:sacCer3localname:sacCer3
fasta:/path/to/my/genomes/sacCer3/sacCer3.faRegex, masking & compressionYou can use a regular expression to filter for matching sequences
(or non-matching sequences by using the-n/--no-matchoption).
For instance, the following command downloads hg38 and saves only the major chromosomes:$genomepyinstallhg38-pUCSC-r'chr[0-9XY]+$'Downloadinggenomefromfromhttp://hgdownload.soe.ucsc.edu/goldenPath/hg38/bigZips/hg38.fa.gz...
Genomedownloadsuccessful,startingpostprocessing...

name:hg38localname:hg38
fasta:/data/genomes/hg38/hg38.fa

$grep">"/data/genomes/hg38/hg38.fa
>chr1
>chr10
>chr11
>chr12
>chr13
>chr14
>chr15
>chr16
>chr17
>chr18
>chr19
>chr2
>chr20
>chr21
>chr22
>chr3
>chr4
>chr5
>chr6
>chr7
>chr8
>chr9
>chrX
>chrYBy default, genome sequences are soft-masked (ACgtN).
Use-m hardfor hard masking (ACNNN), or-m nonefor no masking (ACGTN).$genomepyinstallhg38--maskhardIf you wish to conserve space, you can tell genomepy to compress the downloaded data by passing the-b/--bgzipoption.
SeeConfigurationfor details.$genomepyinstallhg38--bgzipOther providers (any URL/local files)To use assemblies not on NCBI, UCSC, Ensembl or GENCODE, you can give a URL instead of a name, together with--provider URL.
Similarly, if you have a local FASTA file, you can install this using the filepath, together with--provider Local:$genomepyinstall-purlhttps://research.nhgri.nih.gov/hydra/download/assembly/\Hm105_Dovetail_Assembly_1.0.fa.gzThis will install the genome under the filename of the URL/filepath, but can be changed with the-l/--localnameoption.If you add the--annotationflag, genomepy will search the (remote) directory for an annotation file as well.
Should this fail, you can also add a URL to the annotation with--URL-to-annotationwith theURLprovider,
or a filepath with--Local-path-to-annotationwith theLocalprovider:$genomepyinstall-plocal/path/to/genome.fa--Local-path-to-annotation/path/to/gene.annotation.gtfReproducibilityAll selected options are stored in aREADME.txt.
This includes the original name, download location and other genomepy operations (such as regex filtering and time).PluginsPlugins are optional steps that are executed after installing an assembly withgenomepy install.
If you already installed an assembly, you can activate a plugin and rerun the install command.
This will not overwrite your local files, unless you use the--forceoption.Check which plugins are enabled withgenomepy plugin list.Genome blacklistsFor some model organisms, genomepy can download a genome blacklist (generated by theKundaje lab).
Blacklists are only available for these model organisms when downloaded from UCSC, and for the human and mouse genomes.Enable the blacklist plugin to use it:$genomepypluginenableblacklist
Enabledplugins:blacklistAligner indexesYou can also create aligner indexes for several widely used aligners.
Currently, genomepy supports:Bowtie2BWAGMAPHISAT2Minimap2STARThese programs are not installed by genomepy and need to be installed separately for the indexing to work.
The easiest way to do so is with conda, e.g.:conda install -c bioconda bwa starSplice-aware indexing (required for e.g. RNA-seq) can be performed by STAR and Hisat2.
This will be done automatically if the gene annotation was downloaded as well.
Finally, STAR can further improve mapping to (novel) splice junctions by indexing again (see 2-pass mapping mode in the STAR manual).
The second pass is not supported by genomepy.You can configure the index creation withgenomepy plugin enable, e.g.:$genomepypluginenablebwastar
Enabledplugins:blacklist,bwa,starYou can pass the number of threads to use for aligner index creation withgenomepy install --threads(default is 8).ConfigurationAll defaults can be overwritten on the command line and in Python.
However, you can create & edit the config file to change the default settings (full description):$genomepyconfiggenerate
Createdconfigfile~/.config/genomepy/genomepy.yamlGenome locationBy default, genomes will be saved in~/.local/share/genomes.To set the default genome directory, to/data/genomesfor instance,
edit~/.config/genomepy/genomepy.yamland change the following line:genomes_dir:/data/genomesCompressionGenome FASTA files can be stored using bgzip compression.
This means that the FASTA files will take up less space on disk.
Set the following line to your config file:bgzip:TrueMost tools are able to use bgzip-compressed genome files.
One notable exception isbedtools getfasta.
As an alternative, you can use thefaidxcommand-line script frompyfaidxwhich comes installed with genomepy.List available providers$genomepyproviders
GENCODE
Ensembl
UCSC
NCBI
Local
URLList available genomesYou can constrain the genome list by using the-p/--provideroption to search only a specific provider.
Additionally, you can get the absolute--sizeof each genome (this option slows down the search).$genomepygenomes-pUCSC
nameprovideraccessiontax_idannotationspeciesother_infonrek
ailMel1UCSCGCF_000004335.29646✓✗✓✗AiluropodamelanoleucaDec.2009(BGI-Shenzhen1.0/ailMel1)allMis1UCSCGCA_000281125.18496✗✓✗✗AlligatormississippiensisAug.2012(allMis0.2/allMis1)anoCar1UCSCna28377✗✗✓✗AnoliscarolinensisFeb.2007(Broad/anoCar1)Local cache.Note that the first time you rungenomepy searchorlistthe command will take a while as the genome lists have to be downloaded.
The lists are cached locally, which will save time later.
The cached files are stored in~/.cache/genomepyand expire after 7 days (so they stay up to date).
This expiration time can be changed in the config file.
You can also delete this directory to clean the cache usinggenomepy clean.Python APICheck out ourPython API documentation here>>> import genomepy
>>> for row in genomepy.search("GRCh38"):
...    print(row)
...    
['GRCh38.p13', 'Ensembl', 'GCA_000001405.28', 9606, True, 'Homo sapiens', '2014-01-Ensembl/2021-03']
['hg38', 'UCSC', 'GCA_000001405.15', 9606, [True, True, False, True], 'Homo sapiens', 'Dec. 2013 (GRCh38/hg38)']
['GRCh38', 'NCBI', 'GCF_000001405.26', 9606, True, 'Homo sapiens', 'Genome Reference Consortium']
['GRCh38.p1', 'NCBI', 'GCF_000001405.27', 9606, True, 'Homo sapiens', 'Genome Reference Consortium']
['GRCh38.p2', 'NCBI', 'GCF_000001405.28', 9606, True, 'Homo sapiens', 'Genome Reference Consortium']
['GRCh38.p3', 'NCBI', 'GCF_000001405.29', 9606, True, 'Homo sapiens', 'Genome Reference Consortium']

>>> genomepy.install_genome("hg38", annotation=True, provider="UCSC", genomes_dir="./data/genomes")
Downloading genome from UCSC. Target URL: http://hgdownload.soe.ucsc.edu/goldenPath/hg38/bigZips/hg38.fa.gz...
Genome download successful, starting post processing...
name: hg38
local name: hg38
fasta: ./data/genomes/hg38/hg38.fa
Downloading the ncbiRefSeq annotation from the UCSC MySQL database.
Annotation download successful

>>> a = genomepy.Annotation("hg38", genomes_dir="./data/genomes")
>>> a.named_gtf.head(3)
          seqname  ...                                          attribute
gene_name          ...                                                   
DDX11L1      chr1  ...  gene_id "DDX11L1"; transcript_id "NR_046018.2"...
DDX11L1      chr1  ...  gene_id "DDX11L1"; transcript_id "NR_046018.2"...
DDX11L1      chr1  ...  gene_id "DDX11L1"; transcript_id "NR_046018.2"...

>>> start = a.named_gtf.loc["TP63"]["start"].min()
>>> end = a.named_gtf.loc["TP63"]["end"].max()
>>> chrom = a.named_gtf.loc["TP63"]["seqname"][0]

>>> g = genomepy.Genome("hg38", genomes_dir="./data/genomes")
>>> g[chrom][start:end]
>chr3:189596747-189897276
gcaacccgctggggtcaccttccacactgtggaagctttgttcttttgctctttgcagtaaatcttgct...Thegenomepy.Genomeclass builds on top of thepyfaidx.Fastaclass,
see thepyfaidx documentationfor more details.
Thegenomepy.Annotationclass contains pandas Dataframes with GTF and BED files, as well as additional class methods to utilize these.Frequently Asked QuestionsGenomepy utilizes external databases to obtain your files.
Unfortunately this sometimes causes issues.
Here are some of the more common issues, with solutions.Let us know if you encounter issues you cannot solve by creatinga new issue.Provider is offlineOccasionally one of the providers experience connection issues, which can last anywhere between minutes to hours.
When this happens genomepy will warn that the provider appears offline, or that the URL seems broken.If the issue does not pass, you can try to reset genomepy.
Simply rungenomepy cleanon the command line, or rungenomepy.clean()in Python.This genome is missingGenomepy stores provider data on your computer to rerun it faster later.
If a provider was offline during this time, it may miss (parts of) the data.To re-download the data, remove the local data withgenomepy clean, thensearchfor your genome again.This genome is STILL missing/URL is brokenSadly, not everything (naming, structure, filenames) is always consistent on the provider end.
Contact the provider to get it fixed!
One notable group are Ensembl fungi, which seems to be mostly mislabelled.In the meantime, you can still use the power of genomepy by manually retrieving the URLs,
and downloading the files withgenomepy install GENOME_URL -p url --url-to-annotation ANNOTATION_URL.The genomepy config was corruptedYou can create a new one withgenomepy config generateon command line,
orgenomepy.manage_config("generate")in Python.What's genomepy maximum memory usage?Genomepy does not read a genome fully into memory.
Therefore, installing takes less than 1 GB RAM regardless of the genome's size.
Searching NCBI is the most costly operation, using around 3 GB (the first time).Which genome/gene annotation to useEach provider has its pros and cons:Ensembl has excellent gene annotations, but their chromosome names can cause issues with some tools.UCSC has an excellent genome browser, but their gene annotations vary in format.NCBI allows public submissions, and so has the latest versions, although not always complete or error free.Usegenomepy searchto see your options, andgenomepy annotationto check the quality of the gene annotation(s).Getting helpIf you want to report a bug or issue, or have problems with installing or running the software please createa new issue.
This is the preferred way of getting support.
Alternatively, you canmail me.ContributingContributions welcome! Send me a pull request orget in touch.When contributing a PR, please use thedevelopbranch.Quick development setup:Fork & download this repo.cdinto your local repo.git checkout developconda env create -f environment.yamlconda activate genomepypip install -e .git checkout -byour_develop_branchThe command line and python imports will now use the code in your local repo.
To test your changes locally, run the following command:pytest -vvvContributorsSiebren Frölich -@siebrenfMaarten van der Sande -@Maarten-vd-SandeTilman Schäfers@tilschaefSimon van Heeringen -@simonvhDohoon Lee -@dohleeJie Zhu -@alienzjCitationIf you use genomepy in your research, please cite it:10.21105/joss.00320.LicenseThis module is licensed under the terms of theMIT license.