Package: pp-sketchlib
pp-sketchlibLibrary of sketching functions used byPopPUNK. See documentation athttp://poppunk.readthedocs.io/en/latest/sketching.htmlInstallationcondaInstall using conda (recommended):conda install -c conda-forge pp-sketchlibNOTEIf you are gettingUnsatisfiableErroror similar version conflicts try following thetips on conda-forge.
It may also help if you downgrade your version of conda (to 4.5). Installing into
a new environment is recommended.pipOr install through pipYou need to have suitable system dependencies installed.  On ubuntu, this suffices:apt-get update && apt-get install -y --no-install-recommends \
  cmake gfortran libarmadillo-dev libeigen3-dev libopenblas-devThen install pp-sketchlib via pip:pip3 install --user pp-sketchliblocal (build w/ compile)Or install locally:python setup.py installFor this option you will need (all available through conda):a C++14 compiler (GCC >=7.2.0orClang)CMake (>=3.18)pybind11hdf5highfiveEigen(>=v3.0)armadilloIf you wish to compile the GPU code you will also need the CUDA toolkit
installed (tested on 10.2 and 11.0).UsageCreate a set of sketches and save these as a database:sketchlib sketch -l rfiles.txt -o listeria --cpus 4The input filerfiles.txthas one sequence per line. The first column is the sample name, subsequent tab-separated
columns are files containing associated sequences, which may be assemblies or reads, and may be gzipped. For example:sample1    sample1.fa
sample2    sample2.fa
sample3    sample3_1.fq.gz     sample3_2.fq.gzCalculate core and accessory distances between databases withquery dist. If all-vs-all, only the upper triangle is calculated,
for example:sketchlib query dist listeria --cpus 4This will print the distances to STDOUT and can be captured with>. If you wish to output save output files as a database for use with PopPUNK.add the-ooption.Other optionsSketching:--single-strandignores reverse complement k-mers, if input is all in the same sense--min-countminimum k-mer count to include when using reads--exact-counteruses a hash table to count k-mers, which is recommended for non-bacterial datasets.Query:To only use some of the samples in the sketch database, you can add the--subsetoption with a file which lists the required sample names.query jaccardwill output the Jaccard distances, rather than core and accessory distances.query sparsewill output a sparse distance matrix,
using either a--thresholdor the k-nearest (-kNN).Large datasetsWhen working with large datasets, you can increase the--cpusto high numbers and get
a roughly proportional performance increase.For calculating sketches of read datasets, or large numbers of distances, and you have a CUDA compatible GPU,
you can calculate distances on your graphics device even more quickly. Add the--gpuoption with the desired
device ID:sketchlib sketch -l rfiles.txt -o listeria --cpus 4 --gpu 0
sketchlib query dist listeria --gpu 0Both CPU parallelism and the GPU will be used, so be sure to add
both--cpusand--gpufor maximum speed. This is particularly efficient
when sketching reads.BenchmarksSketching 31610 ~3MbL. monocytogenesgenomes takes around 20 minutes.
Calculating all core/accessory distances (500M pairs) takes a further 14 minutes
on a CPU node, or 2 minutes on a GPU. Assigning new queries is twice as fast.ModeParallelisationSpeedSketchingCPU26 genomes per secondRead sketchCPU1.2 genomes per minuteCPU & GPU49 genomes per minuteDistancesCPU170k-1600k distances per secondGPU6000k distances per secondCPU tested using 16 cores on a Intel(R) Xeon(R) Gold 6230 CPU @ 2.10GHz.
GPU tested using an NVIDIA RTX 2080 Ti GPU (4352 CUDA cores @ 1.35GHz).NB: The distance speeds can be increased (linearly) by decreasing number of
k-mers or number of bins. The values above are for eight k-mer lengths
and 10000 bins.NOTEIf your results are slower than this you may wish to compile the library
yourself. The version on conda is optimised for portability over speed,
and we have observed greater speeds compiling with--march=native,
which will automatically be used withpython setup.py install.APIpythonImport the package and call commands. Seepp_sketch/__main__.pyimport pp_sketchlib

pp_sketchlib.constructDatabase(ref_db, names, sequences, kmers, int(round(sketch_size/64)),
strand_preserved, min_count, use_exact, cpus)
distMat = pp_sketchlib.queryDatabase(ref_db, ref_db, rList, qList, kmers,
jaccard, cpus, use_gpu, deviceid)

print(distMat)Available functions:constructDatabase()- sketch genomes and save to a HDF5 database file (returns nothing).queryDatabase()- calculate distances between two database files (returns numpy array).queryDatabaseSparse()- asqueryDatabase(), but only return distances exceeding a
threshold, or nearest neighbours (return is a sparse COO matrix).addRandomToDb()- add a random match calculation to a database (returns nothing).jaccardDist()- Calculate a single jaccard distance between two samples in the same database
(returns a floating point number).squareToLong()- Convert a square distance matrix to long form (returns numpy vector).longToSquare()- Convert a long form distance matrix to a symmetric square distance matrix (returns numpy array).longToSquareMulti()- Converts three long form distance matrices from ref-ref, query-query, query-ref comparisons
into a single square distance matrix (returns a numpy array).sparsifyDists()- Convert a square distance matrix into a sparse matrix, by applying a
distance threshold or number of nearest neighbours (returns a sparse COO matrix).hdf5 database filesRunh5lson a database to see what groups it contains. Databases should always
containsketchand may containrandom. Runh5dumpto see the full contents.
Contents are programmatically accessible with any HDF5 API. See__main__.pyfor an
example in python.Seepoppunk_db_infofrom thePopPUNKpackage for pretty printing.sketchAttributes:sketch_version- version of sketching code used to create the database.
The SHA1 hash of relevant code files (doesn't change with every commit).codon_phased- 1 if codon-phased seeds were used.reverse_complement- 0 if--single-strand.Contains a group for each sample, within each has attributes:base_freq- frequency of A, C, G, T within the input sequence.bbits- bin bits as in bindash (hard-coded as 14).k-mers- k-mer lengths the sketch is at.length- sequence length. Exact if from an assembly, estimated using minhash
if from reads.missing_bases- count of Ns.sketchsize64- number of bins/64, as in bindash.And a dataset for each k-mer length, named as the k-mer length. Each dataset also
has the k-mer length stored as an attribute.randomAttributes:k_max- maximum k-mer length (above this random match chance = 0).k_min- minimum k-mer length (below this will error).use_rc- using both strands?Datasets:centroids- k-means centroids of base frequency clusters.matches_keys- k-mer lengths at which random match chances were calculated.matches_values- random match chances. Flattened matrices in the same order
as the k-mer keys, and row-major across centroid pairs.table_keys- sample order oftable_values.table_values- centroid ID assigned to each sample.AlgorithmsSketchingRead in a sequence to memory. Whether a sequence is reads or not is determinedby the presence of quality scores. Count base composition and number of Ns.Divide the range[0, 2^64)into equally sized bins (number of bins must be a multiple of 64).If assemblies, roll through k-mers at each requested length using ntHash, producing
64-bit hashes.If reads, roll through k-mers as above, but also count occurences and only
pass through those over the minimum count.For each hash, assign it to the appropriate bin, and only store it there if lower than
the current bin value.After completing hashing, keep only the 14 least significant bits in each bin.Apply the optimal densification function, taking values from adjacent bins
iff any bins were not filled.Take blocks of 64 bins, and transpose them into 14 64-bit integers.The array of 64-bit integers is the sketch.DistancesFor each k-mer length, iterate over the two arrays of 64-bit integers being compared.Start with a mask of 64 ON bits.Compute the XOR between the first two 64-bit integers (whether the first of the 14
bin bits of the first 64 bins is identical).Compute the AND between this and the mask, update this as the mask. The mask
thereby keeps track of whether all bin bits in each bin were indentical.After looping over 14 arrays, use POPCNT on the mask to calculate how many of
the first 64 bins were identical.The Jaccard distance is the proportion of identical bins over the total number
of bins.The core and accessory distance is calculated using simple linear regression of log(jaccard)
on k-mer lengths. Core distance isexp(gradient), accessory isexp(intercept).Random match chanceTo create the random match chances:Take the base composition of all samples, and cluster using k-means.For each cluster centroid, create five random genomes using repeated Bernoulli draws
from the base frequencies at the centroid.Choose maximum and minimum k-mer length based on where a Jaccard distance of 0 and 1
would be expected with equal base frequencies.For each k-mer length, at each pairwise combination of centroids (including self),
sketch the random genomes and calculate the jaccard distances.To adjust for random match chance:Assign all samples to their closest k-means centroid by base-composition.Find the pre-calculated random match chance between those two centroids.Downweight the observed Jaccard distance using|obs - random| / (1 - random)If pre-calculated random match chances have not been computed, the formula of
Blais & Blanchette is used (formula 6 in the paper cited below).NotesAll matrix/array structures are row-major, for compatibility with numpy.GPU sketching is only supported for reads. If a mix of reads and assemblies,
sketch each separately and join the databases.GPU sketching filters out any read containing an N, which may give slightly
different results from the CPU code.GPU sketching with variable read lengths is unsupported. Illumina data only for now!GPU distances use lower precision than the CPU code, so slightly different results
are expected.CitationsThe overall method was described in the following paper:Lees JA, Harris SR, Tonkin-Hill G, Gladstone RA, Lo SW, Weiser JN, Corander J, Bentley SD, Croucher NJ. Fast and flexible
bacterial genomic epidemiology with PopPUNK.Genome Research29:1-13 (2019).
doi:10.1101/gr.241455.118This extension uses parts of the following methods, and in some cases their code (license included where required):bindash(written by XiaoFei Zhao):Zhao, X. BinDash, software for fast genome distance estimation on a typical personal laptop.Bioinformatics35:671–673 (2019).doi:10.1093/bioinformatics/bty651ntHash(written by Hamid Mohamadi):Mohamadi, H., Chu, J., Vandervalk, B. P. & Birol, I. ntHash: recursive nucleotide hashing.Bioinformatics32:3492–3494 (2016).doi:10.1093/bioinformatics/btw397countmin(similar to that used in the khmer library, written by the Lab for Data Intensive Biology at UC Davis):Zhang, Q., Pell, J., Canino-Koning, R., Howe, A. C. & Brown, C. T.These are not the k-mers you are looking for: efficient online k-mer counting using a probabilistic data structure.PLoS One 9, e101271 (2014).doi:10.1371/journal.pone.0101271CSRSBlais, E. & Blanchette, M.Common Substrings in Random Strings.Combinatorial Pattern Matching 129–140 (2006).doi:10.1007/11780441_13Building and testing notes (for developers)Runpython setup.py build --debugto build with debug flags onYou can set an environment variableSKETCHLIB_INSTALLto affectpython setup.py:Empty: uses cmakeconda: sets library location to the conda environment, and usessrc/Makefile(used to be used in conda-forge recipe)azure: Usessrc/MakefilecmakeNow requires v3.19. If nvcc version is 11.0 or higher, sm8.6 with device link time optimisation will be used.
Otherwise, code is generated for sm7.0 and sm7.5.makeSeesrc/Makefile. Targets are:all(default): builds test executablessketch_test,matrix_test,read_testandgpu_dist_testpython: builds the python extension, same as cmakeweb: builds the webassembly (requiresemccinstalled and activated)install: installs executables (don't use this)python_install: installs python extensionclean: removes all intermediate build files and executablesModifiers:DEBUG=1runs with debug flagsPROFILE=1runs with profiler flags forncuandnsysGPU=1also build CUDA code (assumes/usr/local/cuda-11.1/and SM v8.6)azureThe repository key for the ubuntu CUDA install is periodically updated, which may cause build failures. Seehttps://developer.nvidia.com/blog/updating-the-cuda-linux-gpg-repository-key/and update inazure-pipelines.yml.Test that Python can build an installable packageBuild a python source package and install it into an empty docker container with vanilla python 3. If this works, then there's a good chance that the version uploaded to pypi will workrm -rf dist
python3 setup.py sdist
docker run --rm -it -v "${PWD}:/src:ro" python:3 /src/docker/installSeethis PRfor the sorts of things we're trying to work around here.Publish to pypiIf things are being weird, the test index can be useful:python3 setup.py sdist
twine upload --repository testpypi dist/*You can test installing this into an empty docker container withdocker run --rm -it --entrypoint bash python:3
apt-get update && apt-get install -y --no-install-recommends \
  cmake gfortran libarmadillo-dev libeigen3-dev libopenblas-dev
pip install -i https://test.pypi.org/simple/ \
  --extra-index-url https://pypi.org/simple \
  pp-sketchlibIt can take a few minutes for the new version to become available so you may want to dopip install -i https://test.pypi.org/simple/ \
  --extra-index-url https://pypi.org/simple \
  pp-sketchlib==1.7.5.3updated with your current version to force installation of the new one.Once satisfied that pip/twine haven't uploaded a completely broken package (and typically once the PR is merged) upload to the main pypi.twine upload dist/*