Package: pyfaidx
DescriptionSamtools provides a function “faidx” (FAsta InDeX), which creates a
small flat index file “.fai” allowing for fast random access to any
subsequence in the indexed FASTA file, while loading a minimal amount of the
file in to memory. This python module implements pure Python classes for
indexing, retrieval, and in-place modification of FASTA files using a samtools
compatible index. The pyfaidx module is API compatible with thepygrseqdb module.
A command-line script “faidx” is installed alongside the pyfaidx module, and
facilitates complex manipulation of FASTA files without any programming knowledge.If you use pyfaidx in your publication, please cite:Shirley MD,Ma Z,Pedersen B,Wheelan S.Efficient “pythonic” access to FASTA files using pyfaidx. PeerJ PrePrints 3:e1196. 2015.InstallationThis package is tested under Linux and macOS using Python 3.7+, and and is available from the PyPI:pip install pyfaidx  # add --user if you don't have rootor download areleaseand:pip install .If usingpip install--usermake sure to add/home/$USER/.local/binto your$PATH(on linux) or/Users/$USER/Library/Python/{pythonversion}/bin(on macOS) if you want to run thefaidxscript.Python 2.6 and 2.7 users may choose to use a package version fromv0.7.2or earier.Usage>>>frompyfaidximportFasta>>>genes=Fasta('tests/data/genes.fasta')>>>genesFasta("tests/data/genes.fasta")# set strict_bounds=True for bounds checkingActs like a dictionary.>>>genes.keys()('AB821309.1','KF435150.1','KF435149.1','NR_104216.1','NR_104215.1','NR_104212.1','NM_001282545.1','NM_001282543.1','NM_000465.3','NM_001282549.1','NM_001282548.1','XM_005249645.1','XM_005249644.1','XM_005249643.1','XM_005249642.1','XM_005265508.1','XM_005265507.1','XR_241081.1','XR_241080.1','XR_241079.1')>>>genes['NM_001282543.1'][200:230]>NM_001282543.1:201-230CTCGTTCCGCGCCCGCCATGGAACCGGATG>>>genes['NM_001282543.1'][200:230].seq'CTCGTTCCGCGCCCGCCATGGAACCGGATG'>>>genes['NM_001282543.1'][200:230].name'NM_001282543.1'# Start attributes are 1-based>>>genes['NM_001282543.1'][200:230].start201# End attributes are 0-based>>>genes['NM_001282543.1'][200:230].end230>>>genes['NM_001282543.1'][200:230].fancy_name'NM_001282543.1:201-230'>>>len(genes['NM_001282543.1'])5466Note that start and end coordinates of Sequence objects are [1, 0]. This can be changed to [0, 0] by passingone_based_attributes=FalsetoFastaorFaidx. This argument only affects theSequence.start/.endattributes, and has no effect on slicing coordinates.Indexes like a list:>>>genes[0][:50]>AB821309.1:1-50ATGGTCAGCTGGGGTCGTTTCATCTGCCTGGTCGTGGTCACCATGGCAACSlices just like a string:>>>genes['NM_001282543.1'][200:230][:10]>NM_001282543.1:201-210CTCGTTCCGC>>>genes['NM_001282543.1'][200:230][::-1]>NM_001282543.1:230-201GTAGGCCAAGGTACCGCCCGCGCCTTGCTC>>>genes['NM_001282543.1'][200:230][::3]>NM_001282543.1:201-230CGCCCCTACA>>>genes['NM_001282543.1'][:]>NM_001282543.1:1-5466CCCCGCCCCT........Slicing start and end coordinates are 0-based, just like Python sequences.Complements and reverse complements just like DNA>>>genes['NM_001282543.1'][200:230].complement>NM_001282543.1(complement):201-230GAGCAAGGCGCGGGCGGTACCTTGGCCTAC>>>genes['NM_001282543.1'][200:230].reverse>NM_001282543.1:230-201GTAGGCCAAGGTACCGCCCGCGCCTTGCTC>>>-genes['NM_001282543.1'][200:230]>NM_001282543.1(complement):230-201CATCCGGTTCCATGGCGGGCGCGGAACGAGFastaobjects can also be accessed using method calls:>>>genes.get_seq('NM_001282543.1',201,210)>NM_001282543.1:201-210CTCGTTCCGC>>>genes.get_seq('NM_001282543.1',201,210,rc=True)>NM_001282543.1(complement):210-201GCGGAACGAGSpliced sequences can be retrieved from a list of [start, end] coordinates:TODOupdate this section# new in v0.5.1segments=[[1,10],[50,70]]>>>genes.get_spliced_seq('NM_001282543.1',segments)>gi|543583786|ref|NM_001282543.1|:1-70CCCCGCCCCTGGTTTCGAGTCGCTGGCCTGCCustom key functions provide cleaner access:>>>frompyfaidximportFasta>>>genes=Fasta('tests/data/genes.fasta',key_function=lambdax:x.split('.')[0])>>>genes.keys()dict_keys(['NR_104212','NM_001282543','XM_005249644','XM_005249645','NR_104216','XM_005249643','NR_104215','KF435150','AB821309','NM_001282549','XR_241081','KF435149','XR_241079','NM_000465','XM_005265508','XR_241080','XM_005249642','NM_001282545','XM_005265507','NM_001282548'])>>>genes['NR_104212'][:10]>NR_104212:1-10CCCCGCCCCTYou can specify a character to split names on, which will generate additional entries:>>>frompyfaidximportFasta>>>genes=Fasta('tests/data/genes.fasta',split_char='.',duplicate_action="first")# default duplicate_action="stop">>>genes.keys()dict_keys(['.1','NR_104212','NM_001282543','XM_005249644','XM_005249645','NR_104216','XM_005249643','NR_104215','KF435150','AB821309','NM_001282549','XR_241081','KF435149','XR_241079','NM_000465','XM_005265508','XR_241080','XM_005249642','NM_001282545','XM_005265507','NM_001282548'])If yourkey_functionorsplit_chargenerates duplicate entries, you can choose what action to take:# new in v0.4.9>>>genes=Fasta('tests/data/genes.fasta',split_char="|",duplicate_action="longest")>>>genes.keys()dict_keys(['gi','563317589','dbj','AB821309.1','','557361099','gb','KF435150.1','557361097','KF435149.1','543583796','ref','NR_104216.1','543583795','NR_104215.1','543583794','NR_104212.1','543583788','NM_001282545.1','543583786','NM_001282543.1','543583785','NM_000465.3','543583740','NM_001282549.1','543583738','NM_001282548.1','530384540','XM_005249645.1','530384538','XM_005249644.1','530384536','XM_005249643.1','530384534','XM_005249642.1','530373237','XM_005265508.1','530373235','XM_005265507.1','530364726','XR_241081.1','530364725','XR_241080.1','530364724','XR_241079.1'])Filter functions (returning True) limit the index:# new in v0.3.8>>>frompyfaidximportFasta>>>genes=Fasta('tests/data/genes.fasta',filt_function=lambdax:x[0]=='N')>>>genes.keys()dict_keys(['NR_104212','NM_001282543','NR_104216','NR_104215','NM_001282549','NM_000465','NM_001282545','NM_001282548'])>>>genes['XM_005249644']KeyError:XM_005249644notintests/data/genes.fasta.Or just get a Python string:>>>frompyfaidximportFasta>>>genes=Fasta('tests/data/genes.fasta',as_raw=True)>>>genesFasta("tests/data/genes.fasta",as_raw=True)>>>genes['NM_001282543.1'][200:230]CTCGTTCCGCGCCCGCCATGGAACCGGATGYou can make sure that you always receive an uppercase sequence, even if your fasta file has lower case>>>frompyfaidximportFasta>>>reference=Fasta('tests/data/genes.fasta.lower',sequence_always_upper=True)>>>reference['gi|557361099|gb|KF435150.1|'][1:70]>gi|557361099|gb|KF435150.1|:2-70TGACATCATTTTCCACCTCTGCTCAGTGTTCAACATCTGACAGTGCTTGCAGGATCTCTCCTGGACAAAYou can also perform line-based iteration, receiving the sequence lines as they appear in the FASTA file:>>>frompyfaidximportFasta>>>genes=Fasta('tests/data/genes.fasta')>>>forlineingenes['NM_001282543.1']:...print(line)CCCCGCCCCTCTGGCGGCCCGCCGTCCCAGACGCGGGAAGAGCTTGGCCGGTTTCGAGTCGCTGGCCTGCAGCTTCCCTGTGGTTTCCCGAGGCTTCCTTGCTTCCCGCTCTGCGAGGAGCCTTTCATCCGAAGGCGGGACGATGCCGGATAATCGGCAGCCGAGGAACCGGCAGCCGAGGATCCGCTCCGGGAACGAGCCTCGTTCCGC...Sequence names are truncated on any whitespace. This is a limitation of the indexing strategy. However, full names can be recovered:# new in v0.3.7>>>frompyfaidximportFasta>>>genes=Fasta('tests/data/genes.fasta')>>>forrecordingenes:...print(record.name)...print(record.long_name)...gi|563317589|dbj|AB821309.1|gi|563317589|dbj|AB821309.1|HomosapiensFGFR2-AHCYL1mRNAforFGFR2-AHCYL1fusionkinaseprotein,completecdsgi|557361099|gb|KF435150.1|gi|557361099|gb|KF435150.1|HomosapiensMDM4proteinvariantY(MDM4)mRNA,completecds,alternativelysplicedgi|557361097|gb|KF435149.1|gi|557361097|gb|KF435149.1|HomosapiensMDM4proteinvariantG(MDM4)mRNA,completecds...# new in v0.4.9>>>frompyfaidximportFasta>>>genes=Fasta('tests/data/genes.fasta',read_long_names=True)>>>forrecordingenes:...print(record.name)...gi|563317589|dbj|AB821309.1|HomosapiensFGFR2-AHCYL1mRNAforFGFR2-AHCYL1fusionkinaseprotein,completecdsgi|557361099|gb|KF435150.1|HomosapiensMDM4proteinvariantY(MDM4)mRNA,completecds,alternativelysplicedgi|557361097|gb|KF435149.1|HomosapiensMDM4proteinvariantG(MDM4)mRNA,completecdsRecords can be accessed efficiently as numpy arrays:# new in v0.5.4>>>frompyfaidximportFasta>>>importnumpyasnp>>>genes=Fasta('tests/data/genes.fasta')>>>np.asarray(genes['NM_001282543.1'])array(['C','C','C',...,'A','A','A'],dtype='|S1')Sequence can be buffered in memory using a read-ahead buffer
for fast sequential access:>>>fromtimeitimporttimeit>>>fetch="genes['NM_001282543.1'][200:230]">>>read_ahead="import pyfaidx; genes = pyfaidx.Fasta('tests/data/genes.fasta', read_ahead=10000)">>>no_read_ahead="import pyfaidx; genes = pyfaidx.Fasta('tests/data/genes.fasta')">>>string_slicing="genes ={}; genes['NM_001282543.1'] = 'N'*10000">>>timeit(fetch,no_read_ahead,number=10000)0.2204863309962093>>>timeit(fetch,read_ahead,number=10000)0.1121859749982832>>>timeit(fetch,string_slicing,number=10000)0.0033553699977346696Read-ahead buffering can reduce runtime by 1/2 for sequential accesses to buffered regions.If you want to modify the contents of your FASTA file in-place, you can use themutableargument.
Any portion of the FastaRecord can be replaced with an equivalent-length string.Warning:This will change the contents of your file immediately and permanently:>>>genes=Fasta('tests/data/genes.fasta',mutable=True)>>>type(genes['NM_001282543.1'])<class'pyfaidx.MutableFastaRecord'>>>>genes['NM_001282543.1'][:10]>NM_001282543.1:1-10CCCCGCCCCT>>>genes['NM_001282543.1'][:10]='NNNNNNNNNN'>>>genes['NM_001282543.1'][:15]>NM_001282543.1:1-15NNNNNNNNNNCTGGCThe FastaVariant class provides a way to integrate single nucleotide variant calls to generate a consensus sequence.# new in v0.4.0>>>consensus=FastaVariant('tests/data/chr22.fasta','tests/data/chr22.vcf.gz',het=True,hom=True)RuntimeWarning:UsingsampleNA06984genotypes.>>>consensus['22'].variant_sites(16042793,21833121,29153196,29187373,29187448,29194610,29821295,29821332,29993842,32330460,32352284)>>>consensus['22'][16042790:16042800]>22:16042791-16042800TCGTAGGACA>>>Fasta('tests/data/chr22.fasta')['22'][16042790:16042800]>22:16042791-16042800TCATAGGACA>>>consensus=FastaVariant('tests/data/chr22.fasta','tests/data/chr22.vcf.gz',sample='NA06984',het=True,hom=True,call_filter='GT == "0/1"')>>>consensus['22'].variant_sites(16042793,29187373,29187448,29194610,29821332)You can also specify paths usingpathlib.Pathobjects.#new in v0.7.1>>>frompyfaidximportFasta>>>frompathlibimportPath>>>genes=Fasta(Path('tests/data/genes.fasta'))>>>genesFasta("tests/data/genes.fasta")Accessing fasta files fromfilesystem_specfilesystems:# new in v0.7.0# pip install fsspec s3fs>>>importfsspec>>>frompyfaidximportFasta>>>of=fsspec.open("s3://broad-references/hg19/v0/Homo_sapiens_assembly19.fasta",anon=True)>>>genes=Fasta(of)It also provides a command-line script:cli script: faidxFetchsequencesfromFASTA.Ifnoregionsarespecified,allentriesintheinputfilearereturned.InputFASTAfilemustbeconsistentlyline-wrapped,andlinewrappingofoutputisbasedoninputlinelengths.positionalarguments:fastaFASTAfileregionsspaceseparatedregionsofsequencetofetche.g.chr1:1-1000optionalarguments:-h,--helpshowthishelpmessageandexit-bBED,--bedBEDbedfileofregions(zero-basedstartcoordinate)-oOUT,--outOUToutputfilename(default:stdout)-i{bed,chromsizes,nucleotide,transposed},--transform{bed,chromsizes,nucleotide,transposed}transformtherequestedregionsintoanotherformat.default:None-c,--complementcomplementthesequence.default:False-r,--reversereversethesequence.default:False-aSIZE_RANGE,--size-rangeSIZE_RANGEselectedsequencesareinthesizerange[low,high].example:1,1000default:None-n,--no-namesomitsequencenamesfromoutput.default:False-f,--full-namesoutputfullnamesincludingdescription.default:False-x,--split-fileswriteeachregiontoaseparatefile(namesarederivedfromregions)-l,--lazyfillin--default-seqformissingranges.default:False-sDEFAULT_SEQ,--default-seqDEFAULT_SEQdefaultbaseformissingpositionsandmasking.default:None-dDELIMITER,--delimiterDELIMITERdelimiterforsplittingnamestomultiplevalues(duplicatenameswillbediscarded).default:None-eHEADER_FUNCTION,--header-functionHEADER_FUNCTIONpythonfunctiontomodifyheaderlinese.g:"lambda x: x.split("|")[0]".default:lambdax:x.split()[0]-u{stop,first,last,longest,shortest},--duplicates-action{stop,first,last,longest,shortest}entrytotakewhenduplicatesequencenamesareencountered.default:stop-gREGEX,--regexREGEXselectedsequencesarethosematchingregularexpression.default:.*-v,--invert-matchselectedsequencesarethosenotmatching'regions'argument.default:False-m,--mask-with-default-seqmasktheFASTAfileusing--default-seqdefault:False-M,--mask-by-casemasktheFASTAfilebychangingtolowercase.default:False-eHEADER_FUNCTION,--header-functionHEADER_FUNCTIONpythonfunctiontomodifyheaderlinese.g:"lambda x: x.split("|")[0]".default:None--no-rebuilddonotrebuildthe.faiindexevenifitisoutofdate.default:False--versionprintpyfaidxversionnumberExamples:$faidx-vtests/data/genes.fasta### Creates an .fai index, but supresses sequence output using --invert-match ###$faidxtests/data/genes.fastaNM_001282543.1:201-210NM_001282543.1:300-320>NM_001282543.1:201-210CTCGTTCCGC>NM_001282543.1:300-320GTAATTGTGTAAGTGACTGCA$faidx--full-namestests/data/genes.fastaNM_001282543.1:201-210>NM_001282543.1|HomosapiensBRCA1associatedRINGdomain1(BARD1),transcriptvariant2,mRNACTCGTTCCGC$faidx--no-namestests/data/genes.fastaNM_001282543.1:201-210NM_001282543.1:300-320CTCGTTCCGCGTAATTGTGTAAGTGACTGCA$faidx--complementtests/data/genes.fastaNM_001282543.1:201-210>NM_001282543.1:201-210(complement)GAGCAAGGCG$faidx--reversetests/data/genes.fastaNM_001282543.1:201-210>NM_001282543.1:210-201CGCCTTGCTC$faidx--reverse--complementtests/data/genes.fastaNM_001282543.1:201-210>NM_001282543.1:210-201(complement)GCGGAACGAG$faidxtests/data/genes.fastaNM_001282543.1>NM_001282543.1:1-5466CCCCGCCCCT..............................................................$faidx--regex"^NM_00128254[35]"genes.fasta>NM_001282543.1......................................................>NM_001282545.1......................................................$faidx--lazytests/data/genes.fastaNM_001282543.1:5460-5480>NM_001282543.1:5460-5480AAAAAAANNNNNNNNNNNNNN$faidx--lazy--default-seq='Q'tests/data/genes.fastaNM_001282543.1:5460-5480>NM_001282543.1:5460-5480AAAAAAAQQQQQQQQQQQQQQ$faidxtests/data/genes.fasta--bedregions.bed...$faidx--transformchromsizestests/data/genes.fastaAB821309.13510KF435150.1481KF435149.1642NR_104216.14573NR_104215.15317NR_104212.15374...$faidx--transformbedtests/data/genes.fastaAB821309.113510KF435150.11481KF435149.11642NR_104216.114573NR_104215.115317NR_104212.115374...$faidx--transformnucleotidetests/data/genes.fastanamestartendATCGNAB821309.1135109557748379440KF435150.114811491201031090KF435149.116422011631291490NR_104216.114573129415528288990NR_104215.1153171567173896810440NR_104212.1153741581175697710600...faidx--transformtransposedtests/data/genes.fastaAB821309.113510ATGGTCAGCTGGGGTCGTTTCATC...KF435150.11481ATGACATCATTTTCCACCTCTGCT...KF435149.11642ATGACATCATTTTCCACCTCTGCT...NR_104216.114573CCCCGCCCCTCTGGCGGCCCGCCG...NR_104215.115317CCCCGCCCCTCTGGCGGCCCGCCG...NR_104212.115374CCCCGCCCCTCTGGCGGCCCGCCG......$faidx--split-filestests/data/genes.fasta$lsAB821309.1.fastaNM_001282549.1.fastaXM_005249645.1.fastaKF435149.1.fastaNR_104212.1.fastaXM_005265507.1.fastaKF435150.1.fastaNR_104215.1.fastaXM_005265508.1.fastaNM_000465.3.fastaNR_104216.1.fastaXR_241079.1.fastaNM_001282543.1.fastaXM_005249642.1.fastaXR_241080.1.fastaNM_001282545.1.fastaXM_005249643.1.fastaXR_241081.1.fastaNM_001282548.1.fastaXM_005249644.1.fasta$faidx--delimiter='_'tests/data/genes.fasta000465.3>000465.3CCCCGCCCCTCTGGCGGCCCGCCGTCCCAGACGCGGGAAGAGCTTGGCCGGTTTCGAGTCGCTGGCCTGCAGCTTCCCTGTGGTTTCCCGAGGCTTCCTTGCTTCCCGCTCTGCGAGGAGCCTTTCATCCGAAGGCGGGA.......$faidx--size-range5500,6000-ichromsizestests/data/genes.fastaNM_000465.35523$faidx-m--bedregions.bedtests/data/genes.fasta### Modifies tests/data/genes.fasta by masking regions using --default-seq character ###$faidx-M--bedregions.bedtests/data/genes.fasta### Modifies tests/data/genes.fasta by masking regions using lowercase characters ###$faidx-e"lambda x: x.split('.')[0]"tests/data/genes.fasta-ibedAB82130913510KF4351501481KF4351491642NR_10421614573NR_10421515317.......Similar syntax assamtools faidxA lower-level Faidx class is also available:>>>frompyfaidximportFaidx>>>fa=Faidx('genes.fa')# can return str with as_raw=True>>>fa.indexOrderedDict([('AB821309.1',IndexRecord(rlen=3510,offset=12,lenc=70,lenb=71)),('KF435150.1',IndexRecord(rlen=481,offset=3585,lenc=70,lenb=71)),...])>>>fa.index['AB821309.1'].rlen3510fa.fetch('AB821309.1',1,10)# these are 1-based genomic coordinates>AB821309.1:1-10ATGGTCAGCTIf the FASTA file is not indexed, whenFaidxis initialized thebuild_indexmethod will automatically run, and
the index will be written to “filename.fa.fai” withwrite_fai().
where “filename.fa” is the original FASTA file.Start and end coordinates are 1-based.Support for compressed FASTApyfaidxcan create and read.faiindices for FASTA files that have
been compressed using thebgziptool fromsamtools.bgzipwrites compressed
data in aBGZFformat.BGZFisgzipcompatible, consisting of
multiple concatenatedgzipblocks, each with an additionalgzipheader making it possible to build an index for rapid random access. I.e.,
files compressed withbgzipare validgzipand so can be read bygunzip.  Seethis descriptionfor more details onbgzip.ChangelogPlease see thereleasesfor a
comprehensive list of version changes.Known issuesI try to fix as many bugs as possible, but most of this work is supported by a single developer. Please check theknown issuesfor bugs relevant to your work. Pull requests are welcome.ContributingCreate a new Pull Request with one feature. If you add a new feature, please
create also the relevant test.To get test running on your machine:Create a new virtualenv and install thedev-requirements.txt.pip install -r dev-requirements.txtDownload the test data running:python tests/data/download_gene_fasta.pyRun the tests withpytestsAcknowledgementsThis project is freely licensed by the author,Matthew
Shirley, and was completed under the
mentorship and financial support of Drs.Sarah
WheelanandVasan
Yegnasubramanianat the Sidney Kimmel
Comprehensive Cancer Center in the Department of Oncology.