Package: kcollections
kcollectionsA fast and efficient library for storing k-mers in c++ and python.Aboutkcollectionsis a python-likedictandsetthat efficiently store kmers as keys or elements of the collection.
It is designed for building/prototyping Bioinformatics tools that rely on kmers but where the includeddictandsetconsume too much memory for use.It implements theBloom Filter Triealgorithm.
This implementation differs from Guillaume et al. by allowing kmers of arbitrary size and by providing a generic dictionary/map data structure for associating arbitrary values with kmers.kcollectionsis currently only available for Python version 2.7.InstallationWe provide some pre-compiled binaries for python 2/3 and Linux and MacOS:pipinstallkcollectionsAlternatively, you can build from source.Build from sourcePrerequisites include:CMake, must be installed manually in order to build the code.boost, at least version 1.65.1uint256_t, included in the repository.pybind11, included in the repository.To build and install the python module from source:gitclonehttps://github.com/masakistan/kcollections.gitcdkcollections# python 3python3setup.pyinstall# orpip3install.# python 2python2setup.pyinstall# orpip2install.Additionally, if you would like to access the functions from a C++ program, you can build static libraries with the following steps:gitclonehttps://github.com/masakistan/kcollections.gitcdkcollections

mkdirbuildcdbuild

cmake..
makeExample UsageUsing KsetSerial Insertion, writing/reading to/from diskKmers can be added one at a time withadd, but the fastest way to add kmers to a set is
to add a DNA sequence usingadd_seq.importkcollectionsks=kcollections.Kset(27)# add single kmerks.add('AAACTGTCTTCCTTTATTTGTTCAGGG')# sequence insertionseq='AAACTGTCTTCCTTTATTTGTTCAGGGATCGTGTCAGTA'ks.add_seq(seq,len(seq))assert'AAACTGTCTTCCTTTATTTGTTCAGGG'inks# iterationforkmerinks:printkmerks.write("data.bs")delks# sometime laterks=Kcollections.Kset()ks.read("data.bs")Parallel InsertionThe fastest way to useKsetis to use multithreaded insertion.
Multithreaded approach is best used when all kmers are loaded upfront.
Kmers not accessible until the threads have been joined usingparallel_add_join.
See the example below on how parallel and serial insertions can be used.importkcollectionsks=kcollections.Kset(27)# multithreaded sequence insertion# nthreads must be a power of 2.# nthreads = 4 or 16 work wellks.parallel_add_init(16)# insert a sequence of kmersks.parallel_add_seq(seq)# insert a single kmerks.parallel_add('AAACTGTCTTCCTTTATTTGTTCACAG')# merge threads together# no parallel add methods can be used after joiningks.parallel_add_join()# serial add is permissible after threads have joinedks.add('AAACTGTCTTCCTTTATTTGTTCACAG')# iterationforkmerinks:print(kmer)printlen(ks)Using KdictSerial Insertionimportkcollectionskd=kcollections.Kdict(str,27)# insertion and value assignmentkd['AAACTGTCTTCCTTTATTTGTTCAGGG']='banana'kd['AAACTGTCTTCCTTTATTTGTTCAGGT']='phone'assertkd['AAACTGTCTTCCTTTATTTGTTCAGGG']=='banana'assertkd['AAACTGTCTTCCTTTATTTGTTCAGGT']=='phone'# iterationforkmer,valinkd.items():print(kmer,val)# removaldelkd['AAACTGTCTTCCTTTATTTGTTCAGGT']Parallel InsertionParallel insertion forKdictrequires the inclusion of a merging function to resolve different values for the same key.
The following snippet adds 27mers from a string of DNA using a provided lambda function to merge value conflicts.
This merge function simply keeps the newest value associated with the kmer.
More examples of merging functions withKdictcan be foundhere.kd=kcollections.Kdict(int,27)kd.parallel_add_init(4)kd.set_merge_func(lambdaprev_val,new_val:new_val)kd.parallel_add_seq(dna,generate_idx(len(dna)))kd.parallel_add_join()Using KcounterKcounteris an implementation of the Python collection'sCounter,
but the keys must be kmers, of course!
LikeKdict, kmers can be added toKcounterone at a time, but the
fastest ways to add kmers to a set is to add an DNA sequence usingadd_seq(orparallel_add_seqfor multithreaded inserts).Serial InsertionfromkcollectionsimportKcounterkc=Kcounter(27)# add single kmerkc['AAACTGTCTTCCTTTATTTGTTCAGGG']+=1# sequence insertionseq='AAACTGTCTTCCTTTATTTGTTCAGGGATCGTGTCAGTA'kc.add_seq(seq)assertkc['AAACTGTCTTCCTTTATTTGTTCAGGG']==2# iterationforkmer,countinkc.items():print(kmer,count)Parallel InsertionfromkcollectionsimportKcounterkc=Kcounter(27)# multithreaded sequence insertion# nthreads must be a power of 2.# nthreads = 4 or 16 work wellkc.parallel_add_init(16)# insert a sequence of kmerskc.parallel_add_seq(seq)# insert a single kmerkc.parallel_add('AAACTGTCTTCCTTTATTTGTTCACAG')# merge threads together# no parallel add methods can be used after joiningkc.parallel_add_join()# updates can be made after the joinkc['AAACTGTCTTCCTTTATTTGTTCAGGG']+=1# iterationforkmer,countinkc.items():print(kmer,count)Performancekcollectionsis quite a bit slower than thedictorsetbut is much more memory-efficient.
We measured memory usage and running time using/usr/bin/time -von aIntel(R) Xeon(R) E5-2650v4 @2.20GHzwith 256 GB RAM.
27mers used for testing were taken from the human genome, no repetitive kmers appear in our dataset providing a worst case scenario where no insertions or queries are pruned before traversing the entire data structure.Memory Usage# kmersksetset1 million25.32 MB105.82 MB10 million63.74 MB906.96 MB100 million0.56 GB11.98 GB500 million2.42 GB48.54 GB1 billion4.43 GB97.07 GB1.5 billion6.44 GB191.61 GB2 billion8.44 GB194.14 GB2.4 billion10.08 GB220.06 GBInsertion TimeInsertion time comparisons using built-in Python set,kcollectionsserial and parallel insert.Read Mapper and AssemblerAn example read mapping algorithm and assembler are provided usingkcollectionsin theapplicationsdirectory.CitationAcknowledgementskcollectionswas built at the Computational Science Laboratory at Brigham Young University by Stanley Fujimoto (@masakistan) and Cole Lyman (@colelyman).Funding was provided by the Utah NASA Space Grant Consortium and the BYU Graduate Research Fellowship.