Package: hammingdist
A small and fast C++ tool to calculate pairwise distances between gene sequences given in fasta format.Python interfaceTo use the Python interface, you should install it from PyPI:python-mpipinstallhammingdistDistances matrixThen, you can e.g. use it in the following way from Python:importhammingdist# To see the different optional arguments available:help(hammingdist.from_fasta)# To import all sequences from a fasta filedata=hammingdist.from_fasta("example.fasta")# To import only the first 100 sequences from a fasta filedata=hammingdist.from_fasta("example.fasta",n=100)# To import all sequences and remove any duplicatesdata=hammingdist.from_fasta("example.fasta",remove_duplicates=True)# To import all sequences from a fasta file, also treating 'X' as a valid characterdata=hammingdist.from_fasta("example.fasta",include_x=True)# The distance data can be accessed point-wise, though looping over all distances might be quite inefficientprint(data[14,42])# The data can be written to disk in csv format (default `distance` Ripser format) and retrieved:data.dump("backup.csv")retrieval=hammingdist.from_csv("backup.csv")# It can also be written in lower triangular format (comma-delimited row-major, `lower-distance` Ripser format):data.dump_lower_triangular("lt.txt")retrieval=hammingdist.from_lower_triangular("lt.txt")# If the `remove_duplicates` option was used, the sequence indices can also be written.# For each input sequence, this prints the corresponding index in the output:data.dump_sequence_indices("indices.txt")# Finally, we can pass the data as a list of strings in Python:data=hammingdist.from_stringlist(["ACGTACGT","ACGTAGGT","ATTTACGT"])DuplicatesWhenfrom_fastais called with the optionremove_duplicates=True, duplicate sequences are removed before constructing the differences matrix.For example given this set of three input sequences:IndexSequence0ACG1ACG2TAGThe distances matrix would be a 2x2 matrix of distances betweenACGandTAG:ACGTAGACG02TAG20The row of the distances matrix corresponding to each index in the original sequence would be:IndexSequenceRow in distances matrix0ACG01ACG02TAT1This last column is what is written to disk byDataSet.dump_sequence_indices.It can also be constructed (as a numpy array) without calculating the distances matrix by usinghammingdist.fasta_sequence_indicesimporthammingdistsequence_indices=hammingdist.fasta_sequence_indices(fasta_file)Maximum distance valuesBy default, the elements in the distances matrix returned byhammingdist.from_fastahave a maximum value of 255.
You can also set a smaller maximum value using themax_distanceargument.
For distances larger than thishammingdist.from_fasta_largesupports distances up to 65535 (but uses twice as much RAM)Distances from reference sequenceThe distance of each sequence in a fasta file from a given reference sequence can be calculated using:importhammingdistdistances=hammingdist.fasta_reference_distances(sequence,fasta_file,include_x=True)This function returns a numpy array that contains the distance of each sequence from the reference sequence.You can also calculate the distance between two individual sequences:importhammingdistdistance=hammingdist.distance("ACGTX","AAGTX",include_x=True)OpenMP on linuxOn linux hammingdist is built with OpenMP (multithreading) support, and will automatically make use of all available CPU threads.CUDA on linuxOn linux hammingdist is also built with CUDA (Nvidia GPU) support.
To use the GPU instead of the CPU, setuse_gpu=Truewhen callingfrom_fasta. Here we also set the maximum distance to 2:importhammingdistdata=hammingdist.from_fasta("example.fasta",use_gpu=True,max_distance=2)Additionally, the lower triangular matrix file can now be directly constructed from the fasta file
using the GPU with thefrom_fasta_to_lower_triangularfunction.
This avoids storing the entire distances matrix in memory and interleaves computation on the GPU with disk I/O on the CPU,
which means it requires less RAM and runs faster.importhammingdisthammingdist.from_fasta_to_lower_triangular('input_fasta.txt','output_lower_triangular.txt',use_gpu=True,max_distance=2)Performance historyA rough measure of the impact of the different performance improvements in hammingdist: