Repository: bacpop/PopPUNK (language: python)

Issue:
Replace dendropy
This will probably work better: https://biopython.org/docs/1.75/api/Bio.Phylo.BaseTree.html#:~:text=of%20the%20outgroup.-,root_at_midpoint,-(self)
New logo
V2 => time for a new logo. Something more poppunk I reckon
Subsetting the database when using poppunk_visualise --cytoscape with --include-files
**Versions**
<!-- Please give the version of PopPUNK and pp-sketchlib used, which can be found be running `poppunk --version` and `poppunk_sketch --version` respectively.
Alternatively, please post the output of `conda list` in your environment -->
poppunk 2.4.0
poppunk_sketch 1.7.4

**Command used and output returned**
<!-- Please give the command you ran, and paste the output from the terminal. Also, describe your data, if appropriate. -->
```
poppunk_visualise --ref-db GPS_v4 --query-db poppunk_clusters --cytoscape --output example_cytoscape --tree none --include-files gps_cluster3_list.txt --network-file GPS_v4/GPS_v4_graph.gt 
```
Output: one .graphml and two .csv files

**Describe the bug**
<!-- Describe the bug, and what you expected to happen. -->
I tried to only include a subset of the dataset in the output with --include-files, which worked fine for the --microreact output. However, the .graphml network contains all isolates from the database rather than just those listed in gps_cluster3_list.txt.

Documentation update
- [x] New sketchlib (v2 API) https://github.com/johnlees/pp-sketchlib/pull/66
- [x] https://github.com/johnlees/PopPUNK/issues/185
- [x] https://github.com/johnlees/PopPUNK/issues/189
- [x] Update reference DBs, and remove danger warning from query assignment page
- [x] Remove PopPIPE danger warning
- [x] Name of `poppunk_info`
Give clearer results when queries are in the reference DB
When using assign mode, if query names match those in the reference database they will simply be omitted from the cluster output. It would be better to do one of the following:

- Give a warning message (easy to miss? may be a lot of them?)
- Always output queries, even if they're in the reference (probably sensible)
- Give an error (same as building a DB)

I'd suggest 1 + 2
Give a 'scaled' option for manual start
Add a scaled: True, scaled: False argument which scales (or not) the provided start and end
HDBSCAN models require consistent version of sklearn to open
Currently trying to install poppunk version 2.3.0.

I used to have a conda installation with poppunk 2.3.0 and poppunk_sketch 1.7.4. I created my own poppunk database to be able to run `poppunk_assign`, ` poppunk --fit-model` and `poppunk_visualise `. Now I need to re-create a conda environment that includes poppunk but it seems that when installing this same version `poppunk_sketch` is not included in the installation, which I believe is needed to run `poppunk_assign`. I attach here a file with the exact error I get when trying to run the commands mentioned above (see poppunk_sketch_error.txt). I have also tried installing other versions in case that would solve the error but I also get an error that it's shown in the other file (see poppunk_2.4_error_message.txt, not sure this is related at all but just in case), I also tried installing it via pip, copying the `poppunk_sketch` exe... but nothing seems to work. I would really appreciate some help in here.

Thanks a lot for the help in advance!

Victoria

[poppunk_2.4_error_message.txt](https://github.com/bacpop/PopPUNK/files/9180281/poppunk_2.4_error_message.txt)
[poppunk_sketch_error.txt](https://github.com/bacpop/PopPUNK/files/9180282/poppunk_sketch_error.txt)


OSError when running poppunk_visualise with cytoscape even in emulated osx-64
**Versions**
poppunk 2.4.0
pp-sketchlib v2.0.0
Python 3.8.13
MacOS Monterey 12.4

**Command used and output returned**
poppunk_visualise 
OSError: error reading from file '':ios_base::clear: unspecified iostream_category error


**Describe the bug**
I was running poppunk_visualise with cytoscape (first without updating the DB)
`poppunk_visualise --ref-db GPS_v4_references --query-db poppunk_clusters_2018 --output visualization_cytoscape__2018 --cytoscape`

but the algorithm stoped with the following error 
`OSError: error reading from file '':ios_base::clear: unspecified iostream_category error`

I don’t know if this could be a problem with my system or is something with the code.
I already run with flag —microreact and with no error.

Then I update by DB  and run
`poppunk_assign --db GPS_v4_references --query query_2018.txt \
--output poppunk_clusters_2018 --qc-filter continue --threads 8 --update-db`

And re-run microreact 
`poppunk_visualise --ref-db poppunk_clusters_2018 --output visualization_microreact_2018 --microreact `  

Again it run. But if using the flag cytoscape it, again, gave the error
`OSError: error reading from file '':ios_base::clear: unspecified iostream_category error`

Is it possible to help me please?
Thank you

--prop-n flag and ambiguous bases
**Versions:**
poppunk 2.4.0
pp-sketchlib 2.0.0

**Command used:**
```
poppunk --create-db --r-files input.tsv --min-k 13 --max-k 29 --k-step 2 --sketch-size 1000000 --min-kmer-count 140 --qc-filter continue --retain-failures --plot-fit 10 --prop-n 0.8 --output minKmer140_sketch1000000 --threads 16
```
**Output returned:**
```
/scicomp/home-pure/ptx4/miniconda3/envs/pp_env/lib/python3.9/site-packages/PopPUNK/sketchlib.py:778: RuntimeWarning: divide b
y zero encountered in log
  args=(klist, np.log(pairwise)),
Fitting k-mer curve failed: Residuals are not finite in the initial point.
With mash input [0.    ,0.0941,0.1367,0.1483,0.1559,0.1681,0.1767,0.1859,0.1952]
Check for low quality input genomes
```
**Output returned:**
```
C166    Ambiguous sequence too high
C167    Ambiguous sequence too high
C168    Ambiguous sequence too high
```

**Describe the bug**
Unclear to me if the flag --prop-n is dealing with the ambiguous bases request as the qc report states 'Ambiguous sequence too high'. I set --prop-n to 80% but still gives same message when I do include --prop-n and when I don't. Additionally, I don't believe most of my data (n =226 out of 251) has that many ambiguous sequences. 
pip installable poppunk, almost
Currently uses the path docker2 because there is historical docker image material there that I now think is redundant/unused?

Pulls in the fix-json-serialisation branch which is unmerged

Was #208, but that used the wrong branch!
Add rapidnj to docker image
Rapidnj installation was added to Dockerfile
pip installable poppunk, almost
Currently uses the path docker2 because there is historical docker image material there that I now think is redundant/unused?

Pulls in the fix-json-serialisation branch which is unmerged

Was #208, but that used the wrong branch!


Discussion / hints:


This should be done by this bit of code which masks the not-included nodes: https://github.com/johnlees/PopPUNK/blob/46aff5d5715b26a7582c733d6956cb4c78748a99/PopPUNK/plot.py#L488-L495

But maybe when we print the graphml it prints the masked nodes too? @nickjcroucher do you remember if this is the case?
All gets saved with the same [function](https://github.com/johnlees/PopPUNK/blob/46aff5d5715b26a7582c733d6956cb4c78748a99/PopPUNK/network.py#L1846) - masking can be a little tricky in graph-tool. Will take a look.
Also @muppi1993 highlighted that running with just `--cytoscape` still generates a tree, which is not needed and quite slow - I think we should change this default, unless there any objections?

Would we need to add a suffix to ensure names remained unique? I would probably err towards three as a default, with 1+2 possible with a flag, but I appreciate that's just more command line complexity. Maybe 1+2 for a standard query, but 3 if the database is being updated?
I also forgot about `--write-references`. That makes it more feasible to error, but offer an option to re-run and override.








Fix patch:
diff --git a/.dockerignore b/.dockerignore
new file mode 100644
index 00000000..86741df0
--- /dev/null
+++ b/.dockerignore
@@ -0,0 +1,6 @@
+build
+dist
+*.egg-info
+.git
+test
+docs
diff --git a/CMakeLists.txt b/CMakeLists.txt
index 6f82f992..57344c9a 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -1,4 +1,4 @@
-cmake_minimum_required(VERSION 3.18)
+cmake_minimum_required(VERSION 3.16)
 project(poppunk_refine)
 set(CMAKE_CXX_STANDARD 14)
 
@@ -25,9 +25,12 @@ if(DEFINED ENV{CONDA_PREFIX})
 endif()
 
 # Add libraries
+include_directories(${CMAKE_CURRENT_SOURCE_DIR}/vendor)
 include_directories(${CMAKE_CURRENT_SOURCE_DIR}/src)
+
+execute_process(COMMAND pybind11-config --cmakedir OUTPUT_STRIP_TRAILING_WHITESPACE OUTPUT_VARIABLE pybind11_DIR)
 find_package(pybind11 REQUIRED)
-find_package (Eigen3 3.3 REQUIRED NO_MODULE)
+find_package(Eigen3 3.3 REQUIRED NO_MODULE)
 #find_package(OpenMP) # This links system openmp if present - conda sorts out rpath but take care
 
 # Define python library target
diff --git a/MANIFEST.in b/MANIFEST.in
index ad9e8edf..4e5b847c 100644
--- a/MANIFEST.in
+++ b/MANIFEST.in
@@ -1,2 +1,5 @@
 recursive-include scripts *.py
-recursive-include PopPUNK/data *.gz
\ No newline at end of file
+recursive-include PopPUNK/data *.gz
+recursive-include vendor *
+recursive-include src *
+include CMakeLists.txt
diff --git a/PopPUNK/__init__.py b/PopPUNK/__init__.py
index 80b46880..5fa591f7 100644
--- a/PopPUNK/__init__.py
+++ b/PopPUNK/__init__.py
@@ -3,9 +3,9 @@
 
 '''PopPUNK (POPulation Partitioning Using Nucleotide Kmers)'''
 
-__version__ = '2.4.6'
+__version__ = '2.5.0'
 
 # Minimum sketchlib version
-SKETCHLIB_MAJOR = 1
-SKETCHLIB_MINOR = 7
+SKETCHLIB_MAJOR = 2
+SKETCHLIB_MINOR = 0
 SKETCHLIB_PATCH = 0
diff --git a/PopPUNK/__main__.py b/PopPUNK/__main__.py
index 2a03340b..b43648ad 100644
--- a/PopPUNK/__main__.py
+++ b/PopPUNK/__main__.py
@@ -7,20 +7,20 @@
 import sys
 # additional
 import numpy as np
-import subprocess
 from collections import defaultdict
 
-import h5py
-
-import pp_sketchlib
-import poppunk_refine
-
 # import poppunk package
 from .__init__ import __version__
 
 # globals
 accepted_weights_types = ["core", "accessory", "euclidean"]
 betweenness_sample_default = 100
+default_length_sigma = 5
+default_prop_n = 0.1
+default_max_zero = 0.05
+default_max_a_dist = 0.5
+default_max_pi_dist = 0.1
+default_max_merge = -1 # off
 
 #******************************#
 #*                            *#
@@ -40,8 +40,12 @@ def get_options():
             help='Create pairwise distances database between reference sequences',
             default=False,
             action='store_true')
+    mode.add_argument('--qc-db',
+            help='Run quality control on a reference database',
+            default=False,
+            action='store_true')
     mode.add_argument('--fit-model',
-            help='Fit a mixture model to a reference database',
+            help='Fit a model to a (QCed) reference database',
             choices=['bgmm', 'dbscan', 'refine', 'lineage', 'threshold'],
             default = False)
     mode.add_argument('--use-model',
@@ -83,24 +87,27 @@ def get_options():
 
     # qc options
     qcGroup = parser.add_argument_group('Quality control options')
-    qcGroup.add_argument('--qc-filter', help='Behaviour following sequence QC step: "stop" [default], "prune"'
-                                                ' (analyse data passing QC), or "continue" (analyse all data)',
-                                                default='stop', type = str, choices=['stop', 'prune', 'continue'])
+    qcGroup.add_argument('--qc-keep', help="Only write failing sequences to a file, don't remove them from the database file",
+                                      default=False, action='store_true')
+    qcGroup.add_argument('--remove-samples', help='A list of names to remove from the database (regardless of any other QC)',
+                                     default=None, type=str)
     qcGroup.add_argument('--retain-failures', help='Retain sketches of genomes that do not pass QC filters in '
-                                                'separate database [default = False]', default=False, action='store_true')
-    qcGroup.add_argument('--max-a-dist', help='Maximum accessory distance to permit [default = 0.5]',
-                                                default = 0.5, type = float)
-    qcGroup.add_argument('--max-pi-dist', help='Maximum core distance to permit [default = 0.5]',
-                                                default = 0.5, type = float)
+                                                   'separate database [default = False]', default=False, action='store_true')
+    qcGroup.add_argument('--max-a-dist', help=f"Maximum accessory distance to permit [default = {default_max_a_dist}]",
+                                                default = default_max_a_dist, type = float)
+    qcGroup.add_argument('--max-pi-dist', help=f"Maximum core distance to permit [default = {default_max_pi_dist}]",
+                                                default = default_max_pi_dist, type = float)
+    qcGroup.add_argument('--max-zero-dist', help=f"Maximum proportion of zero distances to permit [default = {default_max_zero}]",
+                                                default = default_max_zero, type = float)
     qcGroup.add_argument('--type-isolate', help='Isolate from which distances will be calculated for pruning [default = None]',
                                                 default = None, type = str)
-    qcGroup.add_argument('--length-sigma', help='Number of standard deviations of length distribution beyond '
-                                                'which sequences will be excluded [default = 5]', default = 5, type = int)
+    qcGroup.add_argument('--length-sigma', help=f"Number of standard deviations of length distribution beyond "
+                                                f"which sequences will be excluded [default = {default_length_sigma}]", default = default_length_sigma, type = int)
     qcGroup.add_argument('--length-range', help='Allowed length range, outside of which sequences will be excluded '
                                                 '[two values needed - lower and upper bounds]', default=[None,None],
                                                 type = int, nargs = 2)
-    qcGroup.add_argument('--prop-n', help='Threshold ambiguous base proportion above which sequences will be excluded'
-                                                ' [default = 0.1]', default = 0.1,
+    qcGroup.add_argument('--prop-n', help=f"Threshold ambiguous base proportion above which sequences will be excluded"
+                                          f" [default = {default_prop_n}]", default = default_prop_n,
                                                 type = float)
     qcGroup.add_argument('--upper-n', help='Threshold ambiguous base count above which sequences will be excluded',
                                                 default=None, type = int)
@@ -204,8 +211,7 @@ def main():
 
     # Imports are here because graph tool is very slow to load
     from .models import loadClusterFit, BGMMFit, DBSCANFit, RefineFit, LineageFit
-    from .sketchlib import checkSketchlibLibrary
-    from .sketchlib import removeFromDB
+    from .sketchlib import checkSketchlibLibrary, removeFromDB
 
     from .network import construct_network_from_edge_list
     from .network import construct_network_from_assignments
@@ -217,12 +223,11 @@ def main():
     from .plot import writeClusterCsv
     from .plot import plot_scatter
 
-    from .prune_db import prune_distance_matrix
+    from .qc import prune_distance_matrix, qcDistMat, sketchlibAssemblyQC, remove_qc_fail
 
     from .utils import setGtThreads
     from .utils import setupDBFuncs
-    from .utils import readPickle
-    from .utils import qcDistMat
+    from .utils import readPickle, storePickle
     from .utils import createOverallLineage
 
     # check kmer properties
@@ -237,22 +242,8 @@ def main():
         sys.exit(1)
     kmers = np.arange(args.min_k, args.max_k + 1, args.k_step)
 
-    # Dict of QC options for passing to database construction and querying functions
-    qc_dict = {
-        'run_qc': args.create_db,
-        'qc_filter': args.qc_filter,
-        'retain_failures': args.retain_failures,
-        'length_sigma': args.length_sigma,
-        'length_range': args.length_range,
-        'prop_n': args.prop_n,
-        'upper_n': args.upper_n,
-        'max_pi_dist': args.max_pi_dist,
-        'max_a_dist': args.max_a_dist,
-        'type_isolate': args.type_isolate
-    }
-
     # Dict of DB access functions
-    dbFuncs = setupDBFuncs(args, qc_dict)
+    dbFuncs = setupDBFuncs(args)
     createDatabaseDir = dbFuncs['createDatabaseDir']
     constructDatabase = dbFuncs['constructDatabase']
     queryDatabase = dbFuncs['queryDatabase']
@@ -269,13 +260,6 @@ def main():
     # for sketchlib, only supporting a single sketch size
     sketch_sizes = int(round(max(sketch_sizes.values())/64))
 
-    # if a length range is specified, check it makes sense
-    if args.length_range[0] is not None:
-        if args.length_range[0] >= args.length_range[1]:
-            sys.stderr.write('Ensure the specified length range is space-separated argument of'
-                             ' length 2, with the lower value first\n')
-            sys.exit(1)
-
     # check if working with lineages
     if args.fit_model == 'lineage':
         rank_list = sorted([int(x) for x in args.ranks.split(',')])
@@ -284,6 +268,20 @@ def main():
         if max(rank_list) > 100:
             sys.stderr.write("WARNING: Ranks should be small non-zero integers for sensible lineage results\n")
 
+    if args.create_db == False:
+        # Check and set required parameters for other modes
+        if args.ref_db is None:
+            sys.stderr.write("Need to provide --ref-db where .h5 and .dists from "
+                             "--create-db mode were output")
+        if args.distances is None:
+            distances = args.ref_db + "/" + os.path.basename(args.ref_db) + ".dists"
+        else:
+            distances = args.distances
+        if args.output is None:
+            output = args.ref_db
+        else:
+            output = args.output
+
     # run according to mode
     sys.stderr.write("PopPUNK (POPulation Partitioning Using Nucleotide Kmers)\n")
     sys.stderr.write("\t(with backend: " + dbFuncs['backend'] + " v" + dbFuncs['backend_version'] + "\n")
@@ -305,34 +303,26 @@ def main():
 
         # generate sketches and QC sequences to identify sequences not matching specified criteria
         createDatabaseDir(args.output, kmers)
-        seq_names_passing = \
-            constructDatabase(
-                args.r_files,
-                kmers,
-                sketch_sizes,
-                args.output,
-                args.threads,
-                args.overwrite,
-                codon_phased = args.codon_phased,
-                calc_random = True)
+        seq_names = constructDatabase(
+            args.r_files,
+            kmers,
+            sketch_sizes,
+            args.output,
+            args.threads,
+            args.overwrite,
+            codon_phased = args.codon_phased,
+            calc_random = True)
 
         # calculate distances between sequences
-        distMat = queryDatabase(rNames = seq_names_passing,
-                                qNames = seq_names_passing,
+        distMat = queryDatabase(rNames = seq_names,
+                                qNames = seq_names,
                                 dbPrefix = args.output,
                                 queryPrefix = args.output,
                                 klist = kmers,
                                 self = True,
                                 number_plot_fits = args.plot_fit,
                                 threads = args.threads)
-
-        # QC pairwise distances to identify long distances indicative of anomalous sequences in the collection
-        seq_names_passing, distMat = qcDistMat(distMat,
-                                                seq_names_passing,
-                                                seq_names_passing,
-                                                args.output,
-                                                args.output,
-                                                qc_dict)
+        storePickle(seq_names, seq_names, True, distMat, f"{args.output}/{os.path.basename(args.output)}.dists")
 
         # Plot results
         if not args.no_plot:
@@ -340,6 +330,76 @@ def main():
                          args.output + "/" + os.path.basename(args.output) + "_distanceDistribution",
                          args.output + " distances")
 
+    #******************************#
+    #*                            *#
+    #* DB QC                      *#
+    #*                            *#
+    #******************************#
+    if args.qc_db:
+        # if a length range is specified, check it makes sense
+        if args.length_range[0] is not None:
+            if args.length_range[0] >= args.length_range[1]:
+                sys.stderr.write('Ensure the specified length range is space-separated argument of'
+                                ' length 2, with the lower value first\n')
+                sys.exit(1)
+
+        # Dict of QC options for passing to database construction and querying functions
+        qc_dict = {
+            'no_remove': args.qc_keep,
+            'retain_failures': args.retain_failures,
+            'length_sigma': args.length_sigma,
+            'length_range': args.length_range,
+            'prop_n': args.prop_n,
+            'upper_n': args.upper_n,
+            'max_pi_dist': args.max_pi_dist,
+            'max_a_dist': args.max_a_dist,
+            'prop_zero': args.max_zero_dist,
+            'type_isolate': args.type_isolate
+        }
+
+        refList, queryList, self, distMat = readPickle(distances, enforce_self=True)
+
+        fail_unconditionally = {}
+        # Unconditional removal
+        if args.remove_samples:
+            with open(args.remove_samples, 'r') as f:
+                for line in f:
+                    fail_unconditionally[line.rstrip] = ["removed"]
+
+        # assembly qc
+        sys.stderr.write("Running sequence QC\n")
+        pass_assembly_qc, fail_assembly_qc = \
+            sketchlibAssemblyQC(args.ref_db,
+                                refList,
+                                qc_dict)
+        sys.stderr.write(f"{len(fail_assembly_qc)} samples failed\n")
+
+        # QC pairwise distances to identify long distances indicative of anomalous sequences in the collection
+        sys.stderr.write("Running distance QC\n")
+        pass_dist_qc, fail_dist_qc = \
+            qcDistMat(distMat,
+                      refList,
+                      queryList,
+                      args.ref_db,
+                      qc_dict)
+        sys.stderr.write(f"{len(fail_dist_qc)} samples failed\n")
+
+        # Get list of passing samples
+        pass_list = set(refList) - fail_unconditionally.keys() - fail_assembly_qc.keys() - fail_dist_qc.keys()
+        assert(pass_list == set(refList).intersection(set(pass_assembly_qc)).intersection(set(pass_dist_qc)))
+        passed = [x for x in refList if x in pass_list]
+        if qc_dict['type_isolate'] is not None and qc_dict['type_isolate'] not in pass_list:
+            raise RuntimeError('Type isolate ' + qc_dict['type_isolate'] + \
+                               ' not found in isolates after QC; check '
+                               'name of type isolate and QC options\n')
+
+        if len(passed) < len(refList):
+            remove_qc_fail(qc_dict, refList, passed,
+                           [fail_unconditionally, fail_assembly_qc, fail_dist_qc],
+                           args.ref_db, distMat, output,
+                           args.strand_preserved, args.threads)
+
+
     #******************************#
     #*                            *#
     #* model fit and network      *#
@@ -385,11 +445,6 @@ def main():
 
         # Load the distances
         refList, queryList, self, distMat = readPickle(distances, enforce_self=True)
-        seq_names = set(set(refList) | set(queryList))
-        seq_names_passing, distMat = qcDistMat(distMat, refList, queryList, args.ref_db, output, qc_dict)
-        if len(set(seq_names_passing).difference(seq_names)) > 0 and args.qc_filter == "stop":
-            sys.stderr.write("Distances failed quality control (change QC options to run anyway)\n")
-            sys.exit(1)
 
         #******************************#
         #*                            *#
@@ -475,7 +530,6 @@ def main():
             indivNetworks = {}
             lineage_clusters = defaultdict(dict)
             for rank in sorted(rank_list):
-                sys.stderr.write("Network for rank " + str(rank) + "\n")
                 if args.graph_weights:
                     weights = model.edge_weights(rank)
                 else:
@@ -493,6 +547,9 @@ def main():
                                   refList,
                                   printCSV = False,
                                   use_gpu = args.gpu_graph)
+                n_clusters = max(lineage_clusters[rank].values())
+                sys.stderr.write("Network for rank " + str(rank) + " has " +
+                                 str(n_clusters) + " lineages\n")
 
             # print output of each rank as CSV
             overall_lineage = createOverallLineage(rank_list, lineage_clusters)
@@ -553,12 +610,13 @@ def main():
         if model.type != "lineage":
             dist_type_list = ['original']
             dist_string_list = ['']
-            if args.indiv_refine == 'both' or args.indiv_refine == 'core':
-                dist_type_list.append('core')
-                dist_string_list.append('_core')
-            if args.indiv_refine == 'both' or args.indiv_refine == 'accessory':
-                dist_type_list.append('accessory')
-                dist_string_list.append('_accessory')
+            if model.indiv_fitted:
+                if args.indiv_refine == 'both' or args.indiv_refine == 'core':
+                    dist_type_list.append('core')
+                    dist_string_list.append('_core')
+                if args.indiv_refine == 'both' or args.indiv_refine == 'accessory':
+                    dist_type_list.append('accessory')
+                    dist_string_list.append('_accessory')
             # Iterate through different network types
             for dist_type, dist_string in zip(dist_type_list, dist_string_list):
                 if dist_type == 'original':
@@ -572,7 +630,7 @@ def main():
                                         refList,
                                         output,
                                         outSuffix = dist_string,
-                                        type_isolate = qc_dict['type_isolate'],
+                                        type_isolate = args.type_isolate,
                                         threads = args.threads,
                                         use_gpu = args.gpu_graph)
                 nodes_to_remove = set(range(len(refList))).difference(newReferencesIndices)
diff --git a/PopPUNK/assign.py b/PopPUNK/assign.py
index c4228a38..dc2e6464 100644
--- a/PopPUNK/assign.py
+++ b/PopPUNK/assign.py
@@ -3,21 +3,237 @@
 # Copyright 2018-2020 John Lees and Nick Croucher
 
 # universal
+from operator import itemgetter
 import os
 import sys
+import warnings
 # additional
 import numpy as np
-import subprocess
 from collections import defaultdict
-import scipy.optimize
-from scipy.sparse import coo_matrix, bmat, find
+import h5py
 
 # required from v2.1.1 onwards (no mash support)
 import pp_sketchlib
 
 # import poppunk package
 from .__init__ import __version__
-from .models import rankFile
+from .__main__ import default_prop_n, default_length_sigma, default_max_a_dist, \
+    default_max_pi_dist, default_max_zero, default_max_merge
+
+#******************************#
+#*                            *#
+#* Command line parsing       *#
+#*                            *#
+#******************************#
+def get_options():
+
+    import argparse
+
+    parser = argparse.ArgumentParser(description='Assign isolates to strains (by POPulation Partitioning Using Nucleotide Kmers)',
+                                     prog='poppunk_assign')
+
+    # input options
+    iGroup = parser.add_argument_group('Input files')
+    iGroup.add_argument('--db', required=True, type = str, help='Location of built reference database')
+    iGroup.add_argument('--query', required=True, help='File listing query input assemblies')
+    iGroup.add_argument('--distances', help='Prefix of input pickle of pre-calculated distances (if not in --db)')
+    iGroup.add_argument('--external-clustering', help='File with cluster definitions or other labels '
+                                                      'generated with any other method.', default=None)
+
+    # output options
+    oGroup = parser.add_argument_group('Output options')
+    oGroup.add_argument('--output', required=True, help='Prefix for output files (required)')
+    oGroup.add_argument('--plot-fit', help='Create this many plots of some fits relating k-mer to core/accessory distances '
+                                            '[default = 0]', default=0, type=int)
+    oGroup.add_argument('--write-references', help='Write reference database isolates\' cluster assignments out too',
+                                              default=False, action='store_true')
+    oGroup.add_argument('--update-db', help='Update reference database with query sequences', default=False, action='store_true')
+    oGroup.add_argument('--overwrite', help='Overwrite any existing database files', default=False, action='store_true')
+    oGroup.add_argument('--graph-weights', help='Save within-strain Euclidean distances into the graph', default=False, action='store_true')
+
+    # comparison metrics
+    kmerGroup = parser.add_argument_group('Kmer comparison options')
+    kmerGroup.add_argument('--min-kmer-count', default=0, type=int, help='Minimum k-mer count when using reads as input [default = 0]')
+    kmerGroup.add_argument('--exact-count', default=False, action='store_true',
+                           help='Use the exact k-mer counter with reads '
+                                '[default = use countmin counter]')
+    kmerGroup.add_argument('--strand-preserved', default=False, action='store_true',
+                           help='Treat input as being on the same strand, and ignore reverse complement '
+                                'k-mers [default = use canonical k-mers]')
+
+    # qc options
+    qcGroup = parser.add_argument_group('Quality control options for distances')
+    qcGroup.add_argument('--run-qc', help="Run the QC steps", default=False, action="store_true")
+    qcGroup.add_argument('--retain-failures', help='Retain sketches of genomes that do not pass QC filters in '
+                                                'separate database [default = False]', default=False, action='store_true')
+    qcGroup.add_argument('--max-a-dist', help=f"Maximum accessory distance to permit [default = {default_max_a_dist}]",
+                                                default = default_max_a_dist, type = float)
+    qcGroup.add_argument('--max-pi-dist', help=f"Maximum core distance to permit [default = {default_max_pi_dist}]",
+                                                default = default_max_pi_dist, type = float)
+    qcGroup.add_argument('--max-zero-dist', help=f"Maximum proportion of zero distances to permit [default = {default_max_zero}]",
+                                                default = default_max_zero, type = float)
+    qcGroup.add_argument('--max-merge', help=f"Maximum number of cluster merges a sample can cause [default = {default_max_merge}]",
+                                                default = default_max_merge, type = int)
+    qcGroup.add_argument('--betweenness', default=False, action='store_true',
+                               help='Report the betweenness of all the query nodes [default = False]')
+    qcGroup.add_argument('--type-isolate', help='Isolate from which distances can be calculated for pruning [default = None]',
+                                                default = None, type = str)
+    qcGroup.add_argument('--length-sigma', help='Number of standard deviations of length distribution beyond '
+                                                'which sequences will be excluded [default = 5]', default = None, type = int)
+    qcGroup.add_argument('--length-range', help='Allowed length range, outside of which sequences will be excluded '
+                                                '[two values needed - lower and upper bounds]', default=[None,None],
+                                                type = int, nargs = 2)
+    qcGroup.add_argument('--prop-n', help='Threshold ambiguous base proportion above which sequences will be excluded'
+                                                ' [default = None]', default = None,
+                                                type = float)
+    qcGroup.add_argument('--upper-n', help='Threshold ambiguous base count above which sequences will be excluded',
+                                                default=None, type = int)
+
+    # sequence querying
+    queryingGroup = parser.add_argument_group('Database querying options')
+    queryingGroup.add_argument('--serial', default=False, action='store_true',
+                               help='Do assignment one-by-one, not in batches (see docs) [default = False]')
+    queryingGroup.add_argument('--model-dir', help='Directory containing model to use for assigning queries '
+                                                   'to clusters [default = reference database directory]', type = str)
+    queryingGroup.add_argument('--previous-clustering', help='Directory containing previous cluster definitions '
+                                                             'and network [default = use that in the directory '
+                                                             'containing the model]', type = str)
+    queryingGroup.add_argument('--core', help='(with a \'refine\' model) '
+                                                   'Use a core-distance only model for assigning queries '
+                                                   '[default = False]', default=False, action='store_true')
+    queryingGroup.add_argument('--accessory', help='(with a \'refine\' or \'lineage\' model) '
+                                                        'Use an accessory-distance only model for assigning queries '
+                                                        '[default = False]', default=False, action='store_true')
+
+    # processing
+    other = parser.add_argument_group('Other options')
+    other.add_argument('--threads', default=1, type=int, help='Number of threads to use [default = 1]')
+    other.add_argument('--gpu-sketch', default=False, action='store_true', help='Use a GPU when calculating sketches (read data only) [default = False]')
+    other.add_argument('--gpu-dist', default=False, action='store_true', help='Use a GPU when calculating distances [default = False]')
+    other.add_argument('--gpu-graph', default=False, action='store_true', help='Use a GPU when constructing networks [default = False]')
+    other.add_argument('--deviceid', default=0, type=int, help='CUDA device ID, if using GPU [default = 0]')
+    other.add_argument('--version', action='version',
+                       version='%(prog)s '+__version__)
+    other.add_argument('--citation',
+                       action='store_true',
+                       default=False,
+                       help='Give a citation, and possible methods paragraph'
+                            ' based on the command line')
+
+
+    # combine
+    args = parser.parse_args()
+
+    # ensure directories do not have trailing forward slash
+    for arg in [args.db, args.model_dir, args.output, args.previous_clustering]:
+        if arg is not None:
+            arg = arg.rstrip('\\')
+
+    return args
+
+def main():
+    """Main function. Parses cmd line args and runs in the specified mode.
+    """
+
+    #******************************#
+    #*                            *#
+    #* Check command options      *#
+    #*                            *#
+    #******************************#
+    args = get_options()
+
+    # May just want to print the citation
+    if args.citation:
+        from .citation import print_citation
+        print_citation(args, assign=True)
+        sys.exit(0)
+
+    from .sketchlib import checkSketchlibLibrary
+    from .utils import setGtThreads
+    from .utils import setupDBFuncs
+
+    # Dict of QC options for passing to database construction and querying functions
+    if args.run_qc:
+        # define defaults if one QC parameter given
+        # length_sigma
+        if args.length_sigma is not None:
+            length_sigma = args.length_sigma
+        elif None in args.length_range:
+            length_sigma = default_length_sigma
+        else:
+            length_sigma = None
+        # prop_n
+        if args.prop_n is not None:
+            prop_n = args.prop_n
+        elif args.upper_n is None:
+            prop_n = default_prop_n
+        else:
+            prop_n = None
+
+        qc_dict = {
+            'run_qc': True,
+            'retain_failures': args.retain_failures,
+            'length_sigma': length_sigma,
+            'length_range': args.length_range,
+            'prop_n': prop_n,
+            'upper_n': args.upper_n,
+            'max_pi_dist': args.max_pi_dist,
+            'max_a_dist': args.max_a_dist,
+            'prop_zero': args.max_zero_dist,
+            'max_merge': args.max_merge,
+            'betweenness': args.betweenness,
+            'type_isolate': args.type_isolate
+        }
+    else:
+        qc_dict = {'run_qc': False, 'type_isolate': None }
+
+    # Dict of DB access functions for assign_query (which is out of scope)
+    dbFuncs = setupDBFuncs(args)
+
+    sys.stderr.write("PopPUNK: assign\n")
+    sys.stderr.write("\t(with backend: " + dbFuncs['backend'] + " v" + dbFuncs['backend_version'] + "\n")
+    sys.stderr.write('\t sketchlib: ' + checkSketchlibLibrary() + ')\n')
+    sys.stderr.write("Mode: Assigning clusters of query sequences\n\n")
+
+    # Check on parallelisation of graph-tools
+    setGtThreads(args.threads)
+
+    if args.distances is None:
+        distances = args.db + "/" + os.path.basename(args.db) + ".dists"
+    else:
+        distances = args.distances
+
+    #*******************************#
+    #*                             *#
+    #* query assignment            *#
+    #*                             *#
+    #*******************************#
+    assign_query(dbFuncs,
+                 args.db,
+                 args.query,
+                 args.output,
+                 qc_dict,
+                 args.update_db,
+                 args.write_references,
+                 distances,
+                 args.serial,
+                 args.threads,
+                 args.overwrite,
+                 args.plot_fit,
+                 args.graph_weights,
+                 args.model_dir,
+                 args.strand_preserved,
+                 args.previous_clustering,
+                 args.external_clustering,
+                 args.core,
+                 args.accessory,
+                 args.gpu_sketch,
+                 args.gpu_dist,
+                 args.gpu_graph,
+                 args.deviceid,
+                 save_partial_query_graph=False)
+
+    sys.stderr.write("\nDone\n")
 
 #*******************************#
 #*                             *#
@@ -32,13 +248,11 @@ def assign_query(dbFuncs,
                  update_db,
                  write_references,
                  distances,
+                 serial,
                  threads,
                  overwrite,
                  plot_fit,
                  graph_weights,
-                 max_a_dist,
-                 max_pi_dist,
-                 type_isolate,
                  model_dir,
                  strand_preserved,
                  previous_clustering,
@@ -51,6 +265,7 @@ def assign_query(dbFuncs,
                  deviceid,
                  save_partial_query_graph):
     """Code for assign query mode for CLI"""
+    from .qc import sketchlibAssemblyQC
 
     createDatabaseDir = dbFuncs['createDatabaseDir']
     constructDatabase = dbFuncs['constructDatabase']
@@ -60,7 +275,7 @@ def assign_query(dbFuncs,
         sys.stderr.write("--output and --ref-db must be different to "
                          "prevent overwrite.\n")
         sys.exit(1)
-    
+
     # Find distances to reference db
     kmers, sketch_sizes, codon_phased = readDBParams(ref_db)
 
@@ -77,6 +292,18 @@ def assign_query(dbFuncs,
                                 use_gpu = gpu_sketch,
                                 deviceid = deviceid)
 
+    if qc_dict["run_qc"]:
+        pass_assembly_qc, fail_assembly_qc = \
+                sketchlibAssemblyQC(output,
+                                    qNames,
+                                    qc_dict)
+        if len(fail_assembly_qc) > 0:
+            sys.stderr.write(f"{len(fail_assembly_qc)} samples failed:\n"
+                             f"{','.join(fail_assembly_qc.keys())}\n")
+            qNames = [x for x in qNames if x in pass_assembly_qc]
+            if len(qNames) == 0:
+                sys.exit(1)
+
     isolateClustering = assign_query_hdf5(dbFuncs,
                     ref_db,
                     qNames,
@@ -85,25 +312,21 @@ def assign_query(dbFuncs,
                     update_db,
                     write_references,
                     distances,
+                    serial,
                     threads,
                     overwrite,
                     plot_fit,
                     graph_weights,
-                    max_a_dist,
-                    max_pi_dist,
-                    type_isolate,
                     model_dir,
                     strand_preserved,
                     previous_clustering,
                     external_clustering,
                     core,
                     accessory,
-                    gpu_sketch,
                     gpu_dist,
                     gpu_graph,
-                    deviceid,
-                    save_partial_query_graph) 
-    return(isolateClustering)           
+                    save_partial_query_graph)
+    return(isolateClustering)
 
 def assign_query_hdf5(dbFuncs,
                  ref_db,
@@ -113,29 +336,25 @@ def assign_query_hdf5(dbFuncs,
                  update_db,
                  write_references,
                  distances,
+                 serial,
                  threads,
                  overwrite,
                  plot_fit,
                  graph_weights,
-                 max_a_dist,
-                 max_pi_dist,
-                 type_isolate,
                  model_dir,
                  strand_preserved,
                  previous_clustering,
                  external_clustering,
                  core,
                  accessory,
-                 gpu_sketch,
                  gpu_dist,
                  gpu_graph,
-                 deviceid,
                  save_partial_query_graph):
     """Code for assign query mode taking hdf5 as input. Written as a separate function so it can be called
     by web APIs"""
-
     # Modules imported here as graph tool is very slow to load (it pulls in all of GTK?)
-    from .models import loadClusterFit, ClusterFit, BGMMFit, DBSCANFit, RefineFit, LineageFit
+    from tqdm import tqdm
+    from .models import loadClusterFit
 
     from .sketchlib import removeFromDB
 
@@ -146,16 +365,19 @@ def assign_query_hdf5(dbFuncs,
     from .network import printClusters
     from .network import save_network
     from .network import get_vertex_list
+    from .network import printExternalClusters
+    from .network import vertex_betweenness
+
 
     from .plot import writeClusterCsv
 
-    from .prune_db import prune_distance_matrix
+    from .qc import qcDistMat, qcQueryAssignments, prune_distance_matrix, \
+        prune_query_distance_matrix
 
     from .sketchlib import addRandom
 
     from .utils import storePickle
     from .utils import readPickle
-    from .utils import qcDistMat
     from .utils import update_distance_matrices
     from .utils import createOverallLineage
 
@@ -164,7 +386,6 @@ def assign_query_hdf5(dbFuncs,
     readDBParams = dbFuncs['readDBParams']
     getSeqsInDb = dbFuncs['getSeqsInDb']
 
-    sys.stderr.write("Mode: Assigning clusters of query sequences\n\n")
     if ref_db == output:
         sys.stderr.write("--output and --ref-db must be different to "
                          "prevent overwrite.\n")
@@ -172,6 +393,8 @@ def assign_query_hdf5(dbFuncs,
     if (update_db and not distances):
         sys.stderr.write("--update-db requires --distances to be provided\n")
         sys.exit(1)
+    if serial and update_db:
+        raise RuntimeError("--update-db cannot be used with --serial")
 
     # Load the previous model
     model_prefix = ref_db
@@ -181,6 +404,8 @@ def assign_query_hdf5(dbFuncs,
 
     model = loadClusterFit(model_file + '.pkl',
                            model_file + '.npz')
+    if model.type == "lineage" and serial:
+        raise RuntimeError("lineage models cannot be used with --serial")
     model.set_threads(threads)
 
     # Set directories of previous fit
@@ -201,7 +426,7 @@ def assign_query_hdf5(dbFuncs,
             fit_type_list.append('core_refined')
         if accessory:
             fit_type_list.append('accessory_refined')
-    
+
     for fit_type in fit_type_list:
         # Define file name extension
         file_extension_string = ''
@@ -225,7 +450,25 @@ def assign_query_hdf5(dbFuncs,
                 sys.exit(1)
             else:
                 rNames = getSeqsInDb(os.path.join(ref_db, os.path.basename(ref_db) + ".h5"))
-                
+
+        # Deal with name clash
+        same_names = set(rNames).intersection(qNames)
+        if same_names:
+            warnings.warn("Names of queries match names in reference database\n", stacklevel=2)
+            if not write_references:
+                sys.stderr.write("Not running -- change names or add --write-references to override this behaviour\n")
+                sys.exit(1)
+            else:
+                query_db = h5py.File(output + "/" + os.path.basename(output) + ".h5", 'r+')
+                print(output + "/" + os.path.basename(output))
+                sketch_grp = query_db['sketches']
+                for idx, query in enumerate(qNames):
+                    if query in same_names:
+                        new_name = query + "_query"
+                        qNames[idx] = new_name
+                        sketch_grp.move(query, new_name)
+                query_db.close()
+
         if (fit_type == 'default' or (fit_type != 'default' and use_ref_graph)):
             # run query
             qrDistMat = queryDatabase(rNames = rNames,
@@ -237,12 +480,25 @@ def assign_query_hdf5(dbFuncs,
                                       number_plot_fits = plot_fit,
                                       threads = threads,
                                       use_gpu = gpu_dist)
-            
+
         # QC distance matrix
         if qc_dict['run_qc']:
-            seq_names_passing = qcDistMat(qrDistMat, rNames, qNames, ref_db, output, qc_dict)[0]
-        else:
-            seq_names_passing = rNames + qNames
+            sys.stderr.write("Running QC on distance matrix\n")
+            seq_names_passing = \
+                frozenset(qcDistMat(qrDistMat, rNames, qNames, ref_db, qc_dict)[0])
+            failed_samples = frozenset(qNames) - seq_names_passing
+            if len(failed_samples) > 0:
+                sys.stderr.write(f"{len(failed_samples)} samples failed:\n"
+                                 f"{','.join(failed_samples)}\n")
+                if len(failed_samples) == len(qNames):
+                    sys.exit(1)
+                else:
+                    qNames, qrDistMat = \
+                        prune_query_distance_matrix(rNames, qNames, failed_samples, qrDistMat)[0:2]
+                    if update_db:
+                        sys.stderr.write("Queries contained outlier distances, "
+                                         "not updating database\n")
+                        update_db = False
 
         # Load the network based on supplied options
         genomeNetwork, old_cluster_file = \
@@ -253,11 +509,11 @@ def assign_query_hdf5(dbFuncs,
                          core_only = (fit_type == 'core_refined'),
                          accessory_only = (fit_type == 'accessory_refined'),
                          use_gpu = gpu_graph)
-        
+
         if max(get_vertex_list(genomeNetwork, use_gpu = gpu_graph)) != (len(rNames) - 1):
             sys.stderr.write("There are " + str(max(get_vertex_list(genomeNetwork, use_gpu = gpu_graph)) + 1) + \
-                                " vertices in the network but " + str(len(rNames)) + " reference names supplied; " + \
-                                "please check the '--model-dir' variable is pointing to the correct directory\n")
+                             " vertices in the network but " + str(len(rNames)) + " reference names supplied; " + \
+                             "please check the '--model-dir' variable is pointing to the correct directory\n")
 
         if model.type == 'lineage':
             # Assign lineages by calculating query-query information
@@ -318,51 +574,116 @@ def assign_query_hdf5(dbFuncs,
                 queryAssignments = model.assign(qrDistMat, slope = 1)
                 dist_type = 'accessory'
 
+            # QC assignments to check for multi-links
+            if qc_dict['run_qc'] and qc_dict['max_merge'] > 1:
+                sys.stderr.write("Running QC on model assignments\n")
+                seq_names_passing = \
+                    frozenset(qcQueryAssignments(rNames,
+                                                 qNames,
+                                                 queryAssignments,
+                                                 qc_dict['max_merge'],
+                                                 old_cluster_file)[0])
+                failed_samples = frozenset(qNames) - seq_names_passing
+                if len(failed_samples) > 0:
+                    sys.stderr.write(f"{len(failed_samples)} samples failed:\n"
+                                    f"{','.join(failed_samples)}\n")
+                    if len(failed_samples) == len(qNames):
+                        sys.exit(1)
+                    else:
+                        qNames, qrDistMat, queryAssignments = \
+                            prune_query_distance_matrix(rNames, qNames,
+                                failed_samples, qrDistMat, queryAssignments)
+                        if update_db:
+                            sys.stderr.write("Queries contained too many links, "
+                                             "not updating database\n")
+                            update_db = False
+
             # Assign clustering by adding to network
             if graph_weights:
                 weights = qrDistMat
             else:
                 weights = None
 
-            genomeNetwork, qqDistMat = \
-                addQueryToNetwork(dbFuncs,
-                                    rNames,
-                                    qNames,
-                                    genomeNetwork,
-                                    kmers,
-                                    queryAssignments,
-                                    model,
-                                    output,
-                                    distances = distances,
-                                    distance_type = dist_type,
-                                    queryQuery = (update_db and
-                                                    (fit_type == 'default' or
-                                                    (fit_type != 'default' and use_ref_graph)
-                                                    )
-                                                  ),
-                                    strand_preserved = strand_preserved,
-                                    weights = weights,
-                                    threads = threads,
-                                    use_gpu = gpu_graph)
-
             output_fn = os.path.join(output, os.path.basename(output) + file_extension_string)
-            isolateClustering = \
-                {'combined': printClusters(genomeNetwork,
-                                            rNames + qNames,
-                                            output_fn,
-                                            old_cluster_file,
-                                            external_clustering,
-                                            write_references or update_db,
-                                            use_gpu = gpu_graph)}
+            if not serial:
+                genomeNetwork, qqDistMat = \
+                    addQueryToNetwork(dbFuncs,
+                                        rNames,
+                                        qNames,
+                                        genomeNetwork,
+                                        queryAssignments,
+                                        model,
+                                        output,
+                                        kmers = kmers,
+                                        distance_type = dist_type,
+                                        queryQuery = (update_db and
+                                                        (fit_type == 'default' or
+                                                        (fit_type != 'default' and use_ref_graph)
+                                                        )
+                                                    ),
+                                        strand_preserved = strand_preserved,
+                                        weights = weights,
+                                        threads = threads,
+                                        use_gpu = gpu_graph)
+                if qc_dict['run_qc'] and qc_dict['betweenness']:
+                    betweenness = vertex_betweenness(genomeNetwork)[len(rNames):len(rNames) + len(qNames)]
+                    query_betweenness = {query: b for query, b in zip(qNames, betweenness)}
+                    print("query\tbetweenness")
+                    for query, q_betweenness in sorted(query_betweenness.items(), key=itemgetter(1), reverse=True):
+                        print(f"{query}\t{q_betweenness}")
+
+                isolateClustering = \
+                    {'combined': printClusters(genomeNetwork,
+                                                rNames + qNames,
+                                                output_fn,
+                                                old_cluster_file,
+                                                external_clustering,
+                                                write_references or update_db,
+                                                use_gpu = gpu_graph)}
+            else:
+                sys.stderr.write("Assigning serially\n")
+                G_copy = genomeNetwork.copy()
+                isolateClustering = {}
+                for idx, sample in tqdm(enumerate(qNames), total=len(qNames)):
+                    genomeNetwork = \
+                        addQueryToNetwork(dbFuncs,
+                                        rNames,
+                                        [sample],
+                                        genomeNetwork,
+                                        queryAssignments[(idx * len(rNames)):((idx + 1) * len(rNames))],
+                                        model,
+                                        output)[0]
+                    isolate_cluster = printClusters(genomeNetwork,
+                                                rNames + [sample],
+                                                output_fn,
+                                                old_cluster_file,
+                                                external_clustering,
+                                                printRef=False,
+                                                printCSV=False,
+                                                write_unwords=False,
+                                                use_gpu = gpu_graph)
+                    cluster = int(isolate_cluster[sample])
+                    if cluster > len(rNames):
+                        cluster = "novel"
+                    isolateClustering[sample] = cluster
+                    # Reset for next sample
+                    genomeNetwork = G_copy
+
+                # Write out the results
+                cluster_f = open(f"{output}/{os.path.basename(output)}_clusters.csv", 'w')
+                cluster_f.write("Taxon,Cluster\n")
+                for sample, cluster in isolateClustering.items():
+                    cluster_f.write(",".join((sample, str(cluster))) + "\n")
+                cluster_f.close()
+
+                if external_clustering is not None:
+                    printExternalClusters(isolateClustering, external_clustering,
+                                          output, rNames, printRef=False)
+
         # Update DB as requested
         dists_out = output + "/" + os.path.basename(output) + ".dists"
         if update_db:
-            # Check new sequences pass QC before adding them
-            if len(set(seq_names_passing).difference(rNames + qNames)) > 0:
-                sys.stderr.write("Queries contained outlier distances, "
-                                 "not updating database\n")
-            else:
-                sys.stderr.write("Updating reference database to " + output + "\n")
+            sys.stderr.write("Updating reference database to " + output + "\n")
             # Update the network + ref list (everything) - no need to duplicate for core/accessory
             if fit_type == 'default':
                 joinDBs(ref_db, output, output,
@@ -396,8 +717,8 @@ def assign_query_hdf5(dbFuncs,
 
             # Get full distance matrix and save
             complete_distMat = \
-                np.hstack((pp_sketchlib.squareToLong(core_distMat, threads).reshape(-1, 1),
-                           pp_sketchlib.squareToLong(acc_distMat, threads).reshape(-1, 1)))
+                np.hstack((pp_sketchlib.squareToLong(distMat=core_distMat, num_threads=threads).reshape(-1, 1),
+                           pp_sketchlib.squareToLong(distMat=acc_distMat, num_threads=threads).reshape(-1, 1)))
             storePickle(combined_seq, combined_seq, True, complete_distMat, dists_out)
 
             # Copy model if needed
@@ -406,12 +727,12 @@ def assign_query_hdf5(dbFuncs,
 
             # Clique pruning
             if model.type != 'lineage':
-                
+
                 existing_ref_list = []
                 with open(ref_file_name) as refFile:
                     for reference in refFile:
                         existing_ref_list.append(reference.rstrip())
-                
+
                 # Extract references from graph
                 newRepresentativesIndices, newRepresentativesNames, \
                     newRepresentativesFile, genomeNetwork = \
@@ -455,7 +776,7 @@ def assign_query_hdf5(dbFuncs,
                     assert set(postpruning_combined_seq).issuperset(added_references), "Error identifying references"
         else:
             storePickle(rNames, qNames, False, qrDistMat, dists_out)
-            if save_partial_query_graph:
+            if save_partial_query_graph and not serial:
                 if model.type == 'lineage':
                     save_network(genomeNetwork[min(model.ranks)], prefix = output, suffix = '_graph', use_gpu = gpu_graph)
                 else:
@@ -464,216 +785,6 @@ def assign_query_hdf5(dbFuncs,
 
     return(isolateClustering)
 
-#******************************#
-#*                            *#
-#* Command line parsing       *#
-#*                            *#
-#******************************#
-def get_options():
-
-    import argparse
-
-    parser = argparse.ArgumentParser(description='Assign isolates to strains (by POPulation Partitioning Using Nucleotide Kmers)',
-                                     prog='poppunk_assign')
-
-    # input options
-    iGroup = parser.add_argument_group('Input files')
-    iGroup.add_argument('--db', required=True, type = str, help='Location of built reference database')
-    iGroup.add_argument('--query', required=True, help='File listing query input assemblies')
-    iGroup.add_argument('--distances', help='Prefix of input pickle of pre-calculated distances (if not in --db)')
-    iGroup.add_argument('--external-clustering', help='File with cluster definitions or other labels '
-                                                      'generated with any other method.', default=None)
-
-    # output options
-    oGroup = parser.add_argument_group('Output options')
-    oGroup.add_argument('--output', required=True, help='Prefix for output files (required)')
-    oGroup.add_argument('--plot-fit', help='Create this many plots of some fits relating k-mer to core/accessory distances '
-                                            '[default = 0]', default=0, type=int)
-    oGroup.add_argument('--write-references', help='Write reference database isolates\' cluster assignments out too',
-                                              default=False, action='store_true')
-    oGroup.add_argument('--update-db', help='Update reference database with query sequences', default=False, action='store_true')
-    oGroup.add_argument('--overwrite', help='Overwrite any existing database files', default=False, action='store_true')
-    oGroup.add_argument('--graph-weights', help='Save within-strain Euclidean distances into the graph', default=False, action='store_true')
-
-    # comparison metrics
-    kmerGroup = parser.add_argument_group('Kmer comparison options')
-    kmerGroup.add_argument('--min-kmer-count', default=0, type=int, help='Minimum k-mer count when using reads as input [default = 0]')
-    kmerGroup.add_argument('--exact-count', default=False, action='store_true',
-                           help='Use the exact k-mer counter with reads '
-                                '[default = use countmin counter]')
-    kmerGroup.add_argument('--strand-preserved', default=False, action='store_true',
-                           help='Treat input as being on the same strand, and ignore reverse complement '
-                                'k-mers [default = use canonical k-mers]')
-
-    # qc options
-    qcGroup = parser.add_argument_group('Quality control options for distances')
-    qcGroup.add_argument('--qc-filter', help='Behaviour following sequence QC step: "stop" [default], "prune"'
-                                                ' (analyse data passing QC), or "continue" (analyse all data)',
-                                                default='stop', type = str, choices=['stop', 'prune', 'continue'])
-    qcGroup.add_argument('--retain-failures', help='Retain sketches of genomes that do not pass QC filters in '
-                                                'separate database [default = False]', default=False, action='store_true')
-    qcGroup.add_argument('--max-a-dist', help='Maximum accessory distance to permit [default = 0.5]',
-                                                default = 0.5, type = float)
-    qcGroup.add_argument('--max-pi-dist', help='Maximum core distance to permit [default = 0.5]',
-                                                default = 0.5, type = float)
-    qcGroup.add_argument('--type-isolate', help='Isolate from which distances can be calculated for pruning [default = None]',
-                                                default = None, type = str)
-    qcGroup.add_argument('--length-sigma', help='Number of standard deviations of length distribution beyond '
-                                                'which sequences will be excluded [default = 5]', default = None, type = int)
-    qcGroup.add_argument('--length-range', help='Allowed length range, outside of which sequences will be excluded '
-                                                '[two values needed - lower and upper bounds]', default=[None,None],
-                                                type = int, nargs = 2)
-    qcGroup.add_argument('--prop-n', help='Threshold ambiguous base proportion above which sequences will be excluded'
-                                                ' [default = 0.1]', default = None,
-                                                type = float)
-    qcGroup.add_argument('--upper-n', help='Threshold ambiguous base count above which sequences will be excluded',
-                                                default=None, type = int)
-
-    # sequence querying
-    queryingGroup = parser.add_argument_group('Database querying options')
-    queryingGroup.add_argument('--model-dir', help='Directory containing model to use for assigning queries '
-                                                   'to clusters [default = reference database directory]', type = str)
-    queryingGroup.add_argument('--previous-clustering', help='Directory containing previous cluster definitions '
-                                                             'and network [default = use that in the directory '
-                                                             'containing the model]', type = str)
-    queryingGroup.add_argument('--core', help='(with a \'refine\' model) '
-                                                   'Use a core-distance only model for assigning queries '
-                                                   '[default = False]', default=False, action='store_true')
-    queryingGroup.add_argument('--accessory', help='(with a \'refine\' or \'lineage\' model) '
-                                                        'Use an accessory-distance only model for assigning queries '
-                                                        '[default = False]', default=False, action='store_true')
-
-    # processing
-    other = parser.add_argument_group('Other options')
-    other.add_argument('--threads', default=1, type=int, help='Number of threads to use [default = 1]')
-    other.add_argument('--gpu-sketch', default=False, action='store_true', help='Use a GPU when calculating sketches (read data only) [default = False]')
-    other.add_argument('--gpu-dist', default=False, action='store_true', help='Use a GPU when calculating distances [default = False]')
-    other.add_argument('--gpu-graph', default=False, action='store_true', help='Use a GPU when constructing networks [default = False]')
-    other.add_argument('--deviceid', default=0, type=int, help='CUDA device ID, if using GPU [default = 0]')
-    other.add_argument('--version', action='version',
-                       version='%(prog)s '+__version__)
-    other.add_argument('--citation',
-                       action='store_true',
-                       default=False,
-                       help='Give a citation, and possible methods paragraph'
-                            ' based on the command line')
-
-
-    # combine
-    args = parser.parse_args()
-
-    # ensure directories do not have trailing forward slash
-    for arg in [args.db, args.model_dir, args.output, args.previous_clustering]:
-        if arg is not None:
-            arg = arg.rstrip('\\')
-
-    return args
-
-def main():
-    """Main function. Parses cmd line args and runs in the specified mode.
-    """
-
-    #******************************#
-    #*                            *#
-    #* Check command options      *#
-    #*                            *#
-    #******************************#
-    args = get_options()
-
-    # May just want to print the citation
-    if args.citation:
-        from .citation import print_citation
-        print_citation(args, assign=True)
-        sys.exit(0)
-
-    from .sketchlib import checkSketchlibLibrary
-    from .utils import setGtThreads
-    from .utils import setupDBFuncs
-
-    # Dict of QC options for passing to database construction and querying functions
-    if args.length_sigma is None and None in args.length_range and args.prop_n is None \
-        and args.upper_n is None and args.max_a_dist is None and args.max_pi_dist is None:
-        qc_dict = {'run_qc': False, 'type_isolate': None }
-    else:
-        # define defaults if one QC parameter given
-        # length_sigma
-        if args.length_sigma is not None:
-            length_sigma = args.length_sigma
-        elif None in args.length_range:
-            length_sigma = 5 # default used in __main__
-        else:
-            length_sigma = None
-        # prop_n
-        if args.prop_n is not None:
-            prop_n = args.prop_n
-        elif args.upper_n is None:
-            prop_n = 0.1 # default used in __main__
-        else:
-            prop_n = None
-        qc_dict = {
-            'run_qc': True,
-            'qc_filter': args.qc_filter,
-            'retain_failures': args.retain_failures,
-            'length_sigma': length_sigma,
-            'length_range': args.length_range,
-            'prop_n': prop_n,
-            'upper_n': args.upper_n,
-            'max_pi_dist': args.max_pi_dist,
-            'max_a_dist': args.max_a_dist,
-            'type_isolate': args.type_isolate
-        }
-
-    # Dict of DB access functions for assign_query (which is out of scope)
-    dbFuncs = setupDBFuncs(args, qc_dict)
-
-    # run according to mode
-    sys.stderr.write("PopPUNK: assign\n")
-    sys.stderr.write("\t(with backend: " + dbFuncs['backend'] + " v" + dbFuncs['backend_version'] + "\n")
-    sys.stderr.write('\t sketchlib: ' + checkSketchlibLibrary() + ')\n')
-
-    # Check on parallelisation of graph-tools
-    setGtThreads(args.threads)
-
-    if args.distances is None:
-        distances = args.db + "/" + os.path.basename(args.db) + ".dists"
-    else:
-        distances = args.distances
-
-    #*******************************#
-    #*                             *#
-    #* query assignment (function  *#
-    #* at top)                     *#
-    #*                             *#
-    #*******************************#
-    assign_query(dbFuncs,
-                 args.db,
-                 args.query,
-                 args.output,
-                 qc_dict,
-                 args.update_db,
-                 args.write_references,
-                 distances,
-                 args.threads,
-                 args.overwrite,
-                 args.plot_fit,
-                 args.graph_weights,
-                 args.max_a_dist,
-                 args.max_pi_dist,
-                 args.type_isolate,
-                 args.model_dir,
-                 args.strand_preserved,
-                 args.previous_clustering,
-                 args.external_clustering,
-                 args.core,
-                 args.accessory,
-                 args.gpu_sketch,
-                 args.gpu_dist,
-                 args.gpu_graph,
-                 args.deviceid,
-                 save_partial_query_graph=False)
-
-    sys.stderr.write("\nDone\n")
-
 
 if __name__ == '__main__':
     main()
diff --git a/PopPUNK/data/microreact_example.pkl b/PopPUNK/data/microreact_example.pkl
new file mode 100644
index 00000000..ec31f47a
Binary files /dev/null and b/PopPUNK/data/microreact_example.pkl differ
diff --git a/PopPUNK/info.py b/PopPUNK/info.py
index 8a084d87..a86793ab 100644
--- a/PopPUNK/info.py
+++ b/PopPUNK/info.py
@@ -30,7 +30,7 @@
 def get_options():
 
     parser = argparse.ArgumentParser(description='Get information about a PopPUNK database',
-                                     prog='poppunk_db_info')
+                                     prog='poppunk_info')
 
     # input options
     parser.add_argument('--db',
diff --git a/PopPUNK/mandrake.py b/PopPUNK/mandrake.py
new file mode 100644
index 00000000..80f21962
--- /dev/null
+++ b/PopPUNK/mandrake.py
@@ -0,0 +1,177 @@
+#!/usr/bin/env python
+# vim: set fileencoding=<utf-8> :
+# Copyright 2018-2022 John Lees and Nick Croucher
+
+import os
+import sys
+import numpy as np
+from functools import partial
+import random
+
+import pp_sketchlib
+from SCE import wtsne
+try:
+    from SCE import wtsne_gpu_fp64
+    gpu_fn_available = True
+except ImportError:
+    gpu_fn_available = False
+
+from .utils import readPickle
+
+def generate_embedding(seqLabels, accMat, perplexity, outPrefix, overwrite, kNN = 50,
+                       maxIter = 1000000, n_threads = 1, use_gpu = False, device_id = 0):
+    """Generate t-SNE projection using accessory distances
+
+    Writes a plot of t-SNE clustering of accessory distances (.dot)
+
+    Args:
+        seqLabels (list)
+            Processed names of sequences being analysed.
+        accMat (numpy.array)
+            n x n array of accessory distances for n samples.
+        perplexity (int)
+            Perplexity parameter passed to t-SNE
+        outPrefix (str)
+            Prefix for all generated output files, which will be placed in
+            `outPrefix` subdirectory
+        overwrite (bool)
+            Overwrite existing output if present
+            (default = False)
+        kNN (int)
+            Number of neigbours to use with SCE (cannot be > n_samples)
+            (default = 50)
+        maxIter (int)
+            Number of iterations to run
+            (default = 1000000)
+        n_threads (int)
+            Number of CPU threads to use
+            (default = 1)
+        use_gpu (bool)
+            Whether to use GPU libraries
+        device_id (int)
+            Device ID of GPU to be used
+            (default = 0)
+
+    Returns:
+        mandrake_filename (str)
+            Filename with .dot of embedding
+    """
+    # generate accessory genome distance representation
+    mandrake_filename = outPrefix + "/" + os.path.basename(outPrefix) + "_perplexity" + str(perplexity) + "_accessory_mandrake.dot"
+    if os.path.isfile(mandrake_filename) and not overwrite:
+        sys.stderr.write("Mandrake analysis already exists; add --overwrite to replace\n")
+    else:
+        sys.stderr.write("Running mandrake\n")
+        kNN = max(kNN, len(seqLabels) - 1)
+        I, J, dists = pp_sketchlib.sparsifyDists(distMat=accMat, distCutoff=0, kNN=kNN)
+
+        # Set up function call with either CPU or GPU
+        weights = np.ones((len(seqLabels)))
+        random.Random()
+        seed = random.randint(0, 2**32)
+        if use_gpu and gpu_fn_available:
+            sys.stderr.write("Running on GPU\n")
+            n_workers = 65536
+            maxIter = round(maxIter / n_workers)
+            wtsne_call = partial(wtsne_gpu_fp64,
+                                perplexity=perplexity,
+                                maxIter=maxIter,
+                                blockSize=128,
+                                n_workers=n_workers,
+                                nRepuSamp=5,
+                                eta0=1,
+                                bInit=0,
+                                animated=False,
+                                cpu_threads=n_threads,
+                                device_id=device_id,
+                                seed=seed)
+        else:
+            sys.stderr.write("Running on CPU\n")
+            maxIter = round(maxIter / n_threads)
+            wtsne_call = partial(wtsne,
+                                perplexity=perplexity,
+                                maxIter=maxIter,
+                                nRepuSamp=5,
+                                eta0=1,
+                                bInit=0,
+                                animated=False,
+                                n_workers=n_threads,
+                                n_threads=n_threads,
+                                seed=seed)
+
+        # Run embedding with C++ extension
+        embedding_result = wtsne_call(I, J, dists, weights)
+        embedding = np.array(embedding_result.get_embedding()).reshape(-1, 2)
+
+        # print dot file
+        with open(mandrake_filename, 'w') as nFile:
+            nFile.write("graph G { ")
+            for s, seqLabel in enumerate(seqLabels):
+                nFile.write('"' + seqLabel + '"' +
+                            '[x='+str(5*float(embedding[s][0]))+',y='+str(5*float(embedding[s][1]))+']; ')
+            nFile.write("}\n")
+
+    return mandrake_filename
+
+# command line parsing
+def get_options():
+
+    import argparse
+
+    parser = argparse.ArgumentParser(description='Run mandrake embedding of accessory distances',
+                                     prog='poppunk_mandrake')
+
+    # input options
+    parser.add_argument('--distances', required=True, help='Prefix of input pickle and numpy file of pre-calculated '
+                                                           'distances')
+    parser.add_argument('--output', required=True, help='Name of output file')
+    parser.add_argument('--perplexity', help='Perplexity used to generate projection [default = 30]', type=int, default=30)
+    parser.add_argument('--knn', help='Number of neighbours used to generate t-SNE projection [default = 50]', type=int, default=50)
+    parser.add_argument('--iter', help='Number of iterations [default = 1000000]', type=int, default=1000000)
+    parser.add_argument('--cpus', help="Number of CPU threads", type=int, default=1)
+    parser.add_argument('--use-gpu', help='Whether to use GPU libraries for t-SNE calculation', default = False, action='store_true')
+    parser.add_argument('--device-id', help="Device ID of GPU to use", type=int, default=0)
+
+    return parser.parse_args()
+
+
+# main code
+def main():
+
+    # Check input ok
+    args = get_options()
+
+    if not os.path.isdir(args.output):
+        try:
+            os.makedirs(args.output)
+        except OSError:
+            sys.stderr.write("Cannot create output directory\n")
+            sys.exit(1)
+
+    # load saved distance matrix
+    refList, queryList, self, distMat = readPickle(args.distances,
+                                                   enforce_self=True)
+
+    # process list of file names
+    seqLabels = [r.split('/')[-1].split('.')[0] for r in refList]
+
+    # generate accMat
+    accMat = pp_sketchlib.longToSquare(distVec=distMat[:, [1]],
+                                       num_threads=args.cpus)
+
+    # generate accessory genome distance representation
+    generate_embedding(seqLabels,
+                       accMat,
+                       args.perplexity,
+                       args.output,
+                       overwrite=True,
+                       kNN=args.knn,
+                       maxIter=args.iter,
+                       n_threads=args.cpus,
+                       use_gpu = args.use_gpu,
+                       device_id = args.device_id)
+
+if __name__ == "__main__":
+    main()
+
+    sys.exit(0)
diff --git a/PopPUNK/models.py b/PopPUNK/models.py
index 2cdb6f20..f62d7c3e 100644
--- a/PopPUNK/models.py
+++ b/PopPUNK/models.py
@@ -9,13 +9,11 @@
 # additional
 import numpy as np
 import random
-import operator
 import pickle
 import shutil
 import re
 from sklearn import utils
 import scipy.optimize
-from scipy.spatial.distance import euclidean
 from scipy import stats
 import scipy.sparse
 import hdbscan
@@ -770,7 +768,10 @@ def fit(self, X, sample_names, model, max_move, min_move, startFile = None, indi
         # Get starting point
         model.no_scale()
         if startFile:
-            self.mean0, self.mean1 = readManualStart(startFile)
+            self.mean0, self.mean1, scaled = readManualStart(startFile)
+            if not scaled:
+                self.mean0 /= self.scale
+                self.mean1 /= self.scale
         elif model.type == 'dbscan':
             sys.stderr.write("Initial model-based network construction based on DBSCAN fit\n")
             self.mean0 = model.cluster_means[model.within_label, :]
@@ -783,8 +784,9 @@ def fit(self, X, sample_names, model, max_move, min_move, startFile = None, indi
             raise RuntimeError("Unrecognised model type")
 
         # Main refinement in 2D
+        scaled_X = X / self.scale
         self.optimal_x, self.optimal_y, optimal_s = \
-          refineFit(X/self.scale,
+          refineFit(scaled_X,
                     sample_names,
                     self.mean0,
                     self.mean1,
@@ -802,10 +804,12 @@ def fit(self, X, sample_names, model, max_move, min_move, startFile = None, indi
 
         # Output clusters at more positions if requested
         if multi_boundary > 1:
-            multi_refine(X/self.scale,
+            sys.stderr.write("Creating multiple boundary fits\n")
+            multi_refine(scaled_X,
                         sample_names,
                         self.mean0,
                         self.mean1,
+                        self.scale,
                         optimal_s,
                         multi_boundary,
                         self.outPrefix,
@@ -822,7 +826,7 @@ def fit(self, X, sample_names, model, max_move, min_move, startFile = None, indi
                         sys.stderr.write("Refining " + dist_type + " distances separately\n")
                         # optimise core distance boundary
                         core_boundary, accessory_boundary, s = \
-                          refineFit(X/self.scale,
+                          refineFit(scaled_X,
                                     sample_names,
                                     self.mean0,
                                     self.mean1,
@@ -844,7 +848,6 @@ def fit(self, X, sample_names, model, max_move, min_move, startFile = None, indi
                 print(e)
                 sys.stderr.write("Could not separately refine core and accessory boundaries. "
                                  "Using joint 2D refinement only.\n")
-            self.indiv_fitted = True
         y = self.assign(X)
         return y
 
@@ -1081,9 +1084,10 @@ def fit(self, X, accessory):
         for rank in self.ranks:
             row, col, data = \
                 pp_sketchlib.sparsifyDists(
-                    pp_sketchlib.longToSquare(X[:, [self.dist_col]], self.threads),
-                    0,
-                    rank
+                    distMat=pp_sketchlib.longToSquare(distVec=X[:, [self.dist_col]],
+                                                      num_threads=self.threads),
+                    distCutoff=0,
+                    kNN=rank
                 )
             self.__save_sparse__(data, row, col, rank, sample_size, X.dtype)
 
@@ -1199,7 +1203,8 @@ def extend(self, qqDists, qrDists):
             qrDists = cp.array(qrDists)
 
         # Reshape qq and qr dist matrices
-        qqSquare = pp_sketchlib.longToSquare(qqDists[:, [self.dist_col]], self.threads)
+        qqSquare = pp_sketchlib.longToSquare(distVec=qqDists[:, [self.dist_col]],
+                                             num_threads=self.threads)
         qqSquare[qqSquare < epsilon] = epsilon
 
         n_ref = self.nn_dists[self.ranks[0]].shape[0]
diff --git a/PopPUNK/network.py b/PopPUNK/network.py
index 7065ba4c..b8b90c42 100644
--- a/PopPUNK/network.py
+++ b/PopPUNK/network.py
@@ -6,22 +6,16 @@
 # universal
 import os
 import sys
-import re
 # additional
-import glob
 import operator
-import shutil
-import subprocess
 import numpy as np
 import pandas as pd
 from scipy.stats import rankdata
-from tempfile import mkstemp, mkdtemp
 from collections import defaultdict, Counter
 from functools import partial
 from multiprocessing import Pool
 import pickle
 import graph_tool.all as gt
-import dendropy
 
 # Load GPU libraries
 try:
@@ -43,11 +37,7 @@
 from .sketchlib import addRandom
 
 from .utils import iterDistRows
-from .utils import listDistInts
 from .utils import readIsolateTypeFromCsv
-from .utils import readRfile
-from .utils import setupDBFuncs
-from .utils import isolateNameToLabel
 from .utils import check_and_set_gpu
 
 from .unwords import gen_unword
@@ -213,6 +203,7 @@ def cliquePrune(component, graph, reference_indices, components_list):
     """
     if gt.openmp_enabled():
         gt.openmp_set_num_threads(1)
+    sys.setrecursionlimit(3000)
     subgraph = gt.GraphView(graph, vfilt=components_list == component)
     refs = reference_indices.copy()
     if subgraph.num_vertices() <= 2:
@@ -366,14 +357,12 @@ def extractReferences(G, dbOrder, outPrefix, outSuffix = '', type_isolate = None
             gt.openmp_set_num_threads(1)
 
         # Cliques are pruned, taking one reference from each, until none remain
-        sys.setrecursionlimit = 5000
         with Pool(processes=threads) as pool:
             ref_lists = pool.map(partial(cliquePrune,
                                             graph=G,
                                             reference_indices=reference_indices,
                                             components_list=components),
                                  set(components))
-        sys.setrecursionlimit = 1000
         # Returns nested lists, which need to be flattened
         reference_indices = set([entry for sublist in ref_lists for entry in sublist])
 
@@ -588,33 +577,10 @@ def print_network_summary(G, betweenness_sample = betweenness_sample_default, us
                                                    "\tScore (w/ weighted-betweenness)\t\t" + "{:.4f}".format(scores[2])])
                                                    + "\n")
 
-def initial_graph_properties(rlist, qlist):
-    """Initial processing of sequence names for
-    network construction.
-
-    Args:
-        rlist (list)
-            List of reference sequence labels
-        qlist (list)
-            List of query sequence labels
-
-    Returns:
-        vertex_labels (list)
-            Ordered list of sequences in network
-        self_comparison (bool)
-            Whether the network is being constructed from all-v-all distances or
-            reference-v-query information
-    """
-    if rlist == qlist:
-        self_comparison = True
-        vertex_labels = rlist
-    else:
-        self_comparison = False
-        vertex_labels = rlist +  qlist
-    return vertex_labels, self_comparison
 
 def process_weights(distMat, weights_type):
     """Calculate edge weights from the distance matrix
+
     Args:
         distMat (2 column ndarray)
             Numpy array of pairwise distances
@@ -757,7 +723,10 @@ def construct_network_from_edge_list(rlist,
     use_gpu = check_and_set_gpu(use_gpu, gpu_lib, quit_on_fail = True)
 
     # data structures
-    vertex_labels, self_comparison = initial_graph_properties(rlist, qlist)
+    if rlist != qlist:
+        vertex_labels = rlist + qlist
+    else:
+        vertex_labels = rlist
 
     # Create new network
     if use_gpu:
@@ -807,7 +776,9 @@ def construct_network_from_edge_list(rlist,
             if previous_network is not None:
                 for (src, dest) in zip(extra_sources, extra_targets):
                     edge_list.append((src, dest))
-        # build the graph
+
+        # build the graph (from scratch)
+        #TODO append to existing graph
         G = gt.Graph(directed = False)
         G.add_vertex(len(vertex_labels))
         if weights is not None:
@@ -878,7 +849,10 @@ def construct_network_from_df(rlist,
     use_gpu = check_and_set_gpu(use_gpu, gpu_lib, quit_on_fail = True)
 
     # data structures
-    vertex_labels, self_comparison = initial_graph_properties(rlist, qlist)
+    if rlist != qlist:
+        vertex_labels = rlist + qlist
+    else:
+        vertex_labels = rlist
 
     # Check df format is correct
     if weights:
@@ -1024,7 +998,7 @@ def construct_dense_weighted_network(rlist, distMat, weights_type = None, use_gp
     use_gpu = check_and_set_gpu(use_gpu, gpu_lib, quit_on_fail = True)
 
     # data structures
-    vertex_labels, self_comparison = initial_graph_properties(rlist, rlist)
+    vertex_labels = rlist
 
     # Filter weights to only the relevant edges
     if weights is None:
@@ -1243,7 +1217,7 @@ def networkSummary(G, calc_betweenness=True, betweenness_sample = betweenness_sa
                 if size > 3:
                     vfilt = component_assignments.a == component
                     subgraph = gt.GraphView(G, vfilt=vfilt)
-                    betweenness.append(max(gt.betweenness(subgraph, norm = True)[0].a))
+                    betweenness.append(max(vertex_betweenness(subgraph, norm=True)))
                     sizes.append(size)
 
         if len(betweenness) > 1:
@@ -1259,8 +1233,14 @@ def networkSummary(G, calc_betweenness=True, betweenness_sample = betweenness_sa
     scores = [base_score, base_score * (1 - metrics[3]), base_score * (1 - metrics[4])]
     return(metrics, scores)
 
-def addQueryToNetwork(dbFuncs, rList, qList, G, kmers,
-                      assignments, model, queryDB, distances = None, distance_type = 'euclidean',
+# graph-tool only, for now
+def vertex_betweenness(graph, norm=True):
+    """Returns betweenness for nodes in the graph
+    """
+    return gt.betweenness(graph, norm=norm)[0].a
+
+def addQueryToNetwork(dbFuncs, rList, qList, G,
+                      assignments, model, queryDB, kmers = None, distance_type = 'euclidean',
                       queryQuery = False, strand_preserved = False, weights = None, threads = 1,
                       use_gpu = False):
     """Finds edges between queries and items in the reference database,
@@ -1275,8 +1255,6 @@ def addQueryToNetwork(dbFuncs, rList, qList, G, kmers,
             List of query names
         G (graph)
             Network to add to (mutated)
-        kmers (list)
-            List of k-mer sizes
         assignments (numpy.array)
             Cluster assignment of items in qlist
         model (ClusterModel)
@@ -1285,6 +1263,8 @@ def addQueryToNetwork(dbFuncs, rList, qList, G, kmers,
             Query database location
         distances (str)
             Prefix of distance files for extending network
+        kmers (list)
+            List of k-mer sizes
         distance_type (str)
             Distance type to use as weights in network
         queryQuery (bool)
@@ -1310,14 +1290,14 @@ def addQueryToNetwork(dbFuncs, rList, qList, G, kmers,
     # initalise functions
     queryDatabase = dbFuncs['queryDatabase']
 
+    if len(qList) > 1 and kmers is None:
+        raise RuntimeError("Must provide db querying info (kmers) if adding "
+                           "more than one sample, as q-q dists may be needed")
+
     # do not calculate weights unless specified
     if weights is None:
         distance_type = None
 
-    # initialise links data structure
-    new_edges = []
-    assigned = set()
-
     # These are returned
     qqDistMat = None
 
@@ -1336,6 +1316,13 @@ def addQueryToNetwork(dbFuncs, rList, qList, G, kmers,
                                             summarise = False,
                                             use_gpu = use_gpu)
 
+    # Check if any queries were not assigned, run qq dists if so
+    if not queryQuery:
+        edge_count = G.get_total_degrees(list(range(ref_count, ref_count + len(qList))))
+        if np.any(edge_count == 0):
+            sys.stderr.write("Found novel query clusters. Calculating distances between them.\n")
+            queryQuery = True
+
     # Calculate all query-query distances too, if updating database
     if queryQuery:
         if len(qList) == 1:
@@ -1360,70 +1347,14 @@ def addQueryToNetwork(dbFuncs, rList, qList, G, kmers,
                 queryAssignation = model.assign(qqDistMat)
 
             # Add queries to network
-            G = construct_network_from_assignments(qList,
-                                                    qList,
+            vertex_labels = rList + qList
+            G = construct_network_from_assignments(vertex_labels,
+                                                   vertex_labels,
                                                     queryAssignation,
                                                     int_offset = ref_count,
                                                     within_label = model.within_label,
                                                     previous_network = G,
-                                                    old_ids = rList,
-                                                    adding_qq_dists = True,
-                                                    distMat = qqDistMat,
-                                                    weights_type = distance_type,
-                                                    summarise = False,
-                                                    use_gpu = use_gpu)
-
-    # Otherwise only calculate query-query distances for new clusters
-    else:
-        # identify potentially new lineages in list: unassigned is a list of queries with no hits
-        unassigned = set(qList).difference(assigned)
-        query_indices = {k:v+ref_count for v,k in enumerate(qList)}
-        # process unassigned query sequences, if there are any
-        if len(unassigned) > 1:
-            sys.stderr.write("Found novel query clusters. Calculating distances between them.\n")
-
-            # use database construction methods to find links between unassigned queries
-            addRandom(queryDB, qList, kmers, strand_preserved, threads = threads)
-            qqDistMat = queryDatabase(rNames = list(unassigned),
-                                      qNames = list(unassigned),
-                                      dbPrefix = queryDB,
-                                      queryPrefix = queryDB,
-                                      klist = kmers,
-                                      self = True,
-                                      number_plot_fits = 0,
-                                      threads = threads)
-
-            if distance_type == 'core':
-                queryAssignation = model.assign(qqDistMat, slope = 0)
-            elif distance_type == 'accessory':
-                queryAssignation = model.assign(qqDistMat, slope = 1)
-            else:
-                queryAssignation = model.assign(qqDistMat)
-
-            # identify any links between queries and store in the same links dict
-            # links dict now contains lists of links both to original database and new queries
-            # have to use names and link to query list in order to match to node indices
-            for row_idx, (assignment, (query1, query2)) in enumerate(zip(queryAssignation, iterDistRows(qList, qList, self = True))):
-                if assignment == model.within_label:
-                    if weights is not None:
-                        if distance_type == 'core':
-                            dist = weights[row_idx, 0]
-                        elif distance_type == 'accessory':
-                            dist = weights[row_idx, 1]
-                        else:
-                            dist = np.linalg.norm(weights[row_idx, :])
-                        edge_tuple = (query_indices[query1], query_indices[query2], dist)
-                    else:
-                        edge_tuple = (query_indices[query1], query_indices[query2])
-                    new_edges.append(edge_tuple)
-
-            G = construct_network_from_assignments(qList,
-                                                    qList,
-                                                    queryAssignation,
-                                                    int_offset = ref_count,
-                                                    within_label = model.within_label,
-                                                    previous_network = G,
-                                                    old_ids = rList + qList,
+                                                    old_ids = vertex_labels,
                                                     adding_qq_dists = True,
                                                     distMat = qqDistMat,
                                                     weights_type = distance_type,
@@ -1485,6 +1416,8 @@ def printClusters(G, rlist, outPrefix=None, oldClusterFile=None,
     Args:
         G (graph)
             Network used to define clusters
+        rlist (list)
+            Names of samples
         outPrefix (str)
             Prefix for output CSV
             Default = None
diff --git a/PopPUNK/plot.py b/PopPUNK/plot.py
index 99c733a1..95972bd2 100644
--- a/PopPUNK/plot.py
+++ b/PopPUNK/plot.py
@@ -1,5 +1,5 @@
 # vim: set fileencoding=<utf-8> :
-# Copyright 2018-2020 John Lees and Nick Croucher
+# Copyright 2018-2022 John Lees and Nick Croucher
 
 '''Plots of GMM results, k-mer fits, and microreact output'''
 
@@ -12,20 +12,17 @@
 mpl.use('Agg')
 mpl.rcParams.update({'font.size': 18})
 import matplotlib.pyplot as plt
-import matplotlib.lines as lines
 import itertools
 # for other outputs
-from shutil import copyfile
 import pandas as pd
 from collections import defaultdict
-from scipy import spatial
-from sklearn import manifold, utils
+from sklearn import utils
 try:  # sklearn >= 0.22
     from sklearn.neighbors import KernelDensity
 except ImportError:
     from sklearn.neighbors.kde import KernelDensity
 
-from .trees import write_tree, mst_to_phylogeny
+from .trees import write_tree
 
 from .utils import isolateNameToLabel
 from .utils import decisionBoundary
@@ -451,7 +448,7 @@ def drawMST(mst, outPrefix, isolate_clustering, clustering_name, overwrite):
                     output=graph2_file_name, output_size=(3000, 3000))
 
 def outputsForCytoscape(G, G_mst, isolate_names, clustering, outPrefix, epiCsv, queryList = None,
-                        suffix = None, writeCsv = True, viz_subset = None):
+                        suffix = None, writeCsv = True):
     """Write outputs for cytoscape. A graphml of the network, and CSV with metadata
 
     Args:
@@ -476,23 +473,11 @@ def outputsForCytoscape(G, G_mst, isolate_names, clustering, outPrefix, epiCsv,
             (default = None)
         writeCsv (bool)
             Whether to print CSV file to accompany network
-        viz_subset (list)
-            List of sequences to include in visualisation
-
     """
-    
+
     # Avoid circular import
     from .network import save_network
-
-    # mask network if subsetting
-    if viz_subset is not None:
-        viz_vertex = G.new_vertex_property('bool')
-        for n,vertex in enumerate(G.vertices()):
-            if isolate_names[n] in viz_subset:
-                viz_vertex[vertex] = True
-            else:
-                viz_vertex[vertex] = False
-        G.set_vertex_filter(viz_vertex)
+    import graph_tool.all as gt
 
     # edit names
     seqLabels = isolateNameToLabel(isolate_names)
@@ -507,6 +492,18 @@ def outputsForCytoscape(G, G_mst, isolate_names, clustering, outPrefix, epiCsv,
         suffix = suffix + '_cytoscape'
     save_network(G, prefix = outPrefix, suffix = suffix, use_graphml = True)
 
+    # Save each component too (useful for very large graphs)
+    component_assignments, component_hist = gt.label_components(G)
+    for component_idx in range(len(component_hist)):
+        remove_list = []
+        for vidx, v_component in enumerate(component_assignments.a):
+            if v_component != component_idx:
+                remove_list.append(vidx)
+        G_copy = G.copy()
+        G_copy.remove_vertex(remove_list)
+        save_network(G_copy, prefix = outPrefix, suffix = "_component_" + str(component_idx + 1), use_graphml = True)
+        del G_copy
+
     if G_mst != None:
         isolate_labels = isolateNameToLabel(G_mst.vp.id)
         for n,v in enumerate(G_mst.vertices()):
@@ -596,10 +593,10 @@ def writeClusterCsv(outfile, nodeNames, nodeLabels, clustering,
 
     # process epidemiological data without duplicating names
     # used by PopPUNK
-    columns_to_be_omitted = ['id', 'Id', 'ID', 'combined_Cluster__autocolour',
-    'core_Cluster__autocolour', 'accessory_Cluster__autocolour',
-    'overall_Lineage']
     if epiCsv is not None:
+        columns_to_be_omitted = ['id', 'Id', 'ID', 'combined_Cluster__autocolour',
+        'core_Cluster__autocolour', 'accessory_Cluster__autocolour',
+        'overall_Lineage']
         epiData = pd.read_csv(epiCsv, index_col = False, quotechar='"')
         epiData.index = isolateNameToLabel(epiData.iloc[:,0])
         for e in epiData.columns.values:
@@ -687,8 +684,8 @@ def writeClusterCsv(outfile, nodeNames, nodeLabels, clustering,
         sys.exit(1)
 
 def outputsForMicroreact(combined_list, clustering, nj_tree, mst_tree, accMat, perplexity,
-                         outPrefix, epiCsv, queryList = None, overwrite = False,
-                         use_gpu = False):
+                         outPrefix, epiCsv, queryList = None, overwrite = False, n_threads = 1,
+                         use_gpu = False, device_id = 0):
     """Generate files for microreact
 
     Output a neighbour joining tree (.nwk) from core distances, a plot of t-SNE clustering
@@ -718,29 +715,108 @@ def outputsForMicroreact(combined_list, clustering, nj_tree, mst_tree, accMat, p
             (default = None)
         overwrite (bool)
             Overwrite existing output if present (default = False)
+        n_threads (int)
+            Number of CPU threads to use
+            (default = 1)
         use_gpu (bool)
             Whether to use a GPU for t-SNE generation
+        device_id (int)
+            Device ID of GPU to be used
+            (default = 0)
+    Returns:
+        outfiles (list)
+            List of output files create
     """
     # Avoid recursive import
-    from .tsne import generate_tsne
+    from .mandrake import generate_embedding
 
     # generate sequence labels
     seqLabels = isolateNameToLabel(combined_list)
 
     # check CSV before calculating other outputs
+    outfiles = [outPrefix + "/" + os.path.basename(outPrefix) + "_microreact_clusters.csv"]
     writeClusterCsv(outPrefix + "/" + os.path.basename(outPrefix) + "_microreact_clusters.csv",
                         combined_list, combined_list, clustering, 'microreact', epiCsv, queryList)
 
     # write the phylogeny .nwk; t-SNE network .dot; clusters + data .csv
-    generate_tsne(seqLabels, accMat, perplexity, outPrefix, overwrite, use_gpu)
+    embedding_file = generate_embedding(seqLabels, accMat, perplexity, outPrefix, overwrite,
+                       kNN=100, maxIter=1000000, n_threads=n_threads,
+                       use_gpu=use_gpu, device_id=device_id)
+    outfiles.append(embedding_file)
 
     # write NJ tree
     if nj_tree is not None:
         write_tree(nj_tree, outPrefix, "_core_NJ.nwk", overwrite)
+        outfiles.append(outPrefix + "/" + os.path.basename(outPrefix) + "_core_NJ.nwk")
 
     # write MST
     if mst_tree is not None:
         write_tree(mst_tree, outPrefix, "_MST.nwk", overwrite)
+        outfiles.append(outPrefix + "/" + os.path.basename(outPrefix) + "_MST.nwk")
+
+    return outfiles
+
+def createMicroreact(prefix, microreact_files, api_key=None):
+    """Creates a .microreact file, and instance via the API
+
+    Args:
+        prefix (str)
+            Prefix for output file
+        microreact_files (str)
+            List of Microreact files [clusters, dot, tree, mst_tree]
+        api_key (str)
+            API key for your account
+    """
+    import pkg_resources
+    import pickle
+    import requests
+    import json
+    from datetime import datetime
+
+    microreact_api_new_url = "https://microreact.org/api/projects/create"
+    description_string = "PopPUNK run on " + datetime.now().strftime("%Y-%b-%d %H:%M")
+    # Load example JSON to be modified
+    with pkg_resources.resource_stream(__name__, 'data/microreact_example.pkl') as example_pickle:
+        json_pickle = pickle.load(example_pickle)
+    json_pickle["meta"]["name"] = description_string
+
+    # Read data in
+    with open(microreact_files[0]) as cluster_file:
+        csv_string = cluster_file.read()
+        json_pickle["files"]["data-file-1"]["blob"] = csv_string
+    with open(microreact_files[1], 'r') as dot_file:
+        dot_string = dot_file.read()
+        json_pickle["files"]["network-file-1"] = {"id": "network-file-1",
+                                                  "name": "network.dot",
+                                                  "format": "text/vnd.graphviz",
+                                                  "blob": dot_string}
+        json_pickle["networks"]["network-1"] = {"title": "Network",
+                                                "file": "network-file-1",
+                                                "nodeField": "id"}
+    if len(microreact_files) > 2:
+        with open(microreact_files[2], 'r') as tree_file:
+            tree_string = tree_file.read()
+            json_pickle["files"]["tree-file-1"]["blob"] = tree_string
+    else:
+        del json_pickle["files"]["tree-file-1"]
+
+    with open(prefix + "/" + os.path.basename(prefix) + ".microreact", 'w') as json_file:
+        json.dump(json_pickle, json_file)
+
+    url = None
+    if api_key != None:
+        headers = {"Content-type": "application/json; charset=UTF-8",
+                   "Access-Token": api_key}
+        r = requests.post(microreact_api_new_url, data=json.dumps(json_pickle), headers=headers)
+        if not r.ok:
+            if r.status_code == 400:
+                sys.stderr.write("Microreact API call failed with response " + r.text + "\n")
+            else:
+                sys.stderr.write("Microreact API call failed with unknown response code " + str(r.status_code) + "\n")
+        else:
+            url = r.json()['url']
+
+    return url
 
 def outputsForPhandango(combined_list, clustering, nj_tree, mst_tree, outPrefix, epiCsv,
                         queryList = None, overwrite = False):
@@ -772,9 +848,6 @@ def outputsForPhandango(combined_list, clustering, nj_tree, mst_tree, outPrefix,
         threads (int)
             Number of threads to use with rapidnj
     """
-    # generate sequence labels
-    seqLabels = isolateNameToLabel(combined_list)
-
     # print clustering file
     writeClusterCsv(outPrefix + "/" + os.path.basename(outPrefix) + "_phandango_clusters.csv",
                     combined_list, combined_list, clustering, 'phandango', epiCsv, queryList)
@@ -815,9 +888,6 @@ def outputsForGrapetree(combined_list, clustering, nj_tree, mst_tree, outPrefix,
         overwrite (bool)
             Overwrite existing output if present (default = False).
     """
-    # generate sequence labels
-    seqLabels = isolateNameToLabel(combined_list)
-
     # print clustering file
     writeClusterCsv(outPrefix + "/" + os.path.basename(outPrefix) + "_grapetree_clusters.csv",
                     combined_list, combined_list, clustering, 'grapetree', epiCsv, queryList)
diff --git a/PopPUNK/prune_db.py b/PopPUNK/prune_db.py
deleted file mode 100755
index 22b17888..00000000
--- a/PopPUNK/prune_db.py
+++ /dev/null
@@ -1,161 +0,0 @@
-#!/usr/bin/env python
-# vim: set fileencoding=<utf-8> :
-# Copyright 2018-2020 John Lees and Nick Croucher
-
-# universal
-import os
-import sys
-# additional
-import numpy as np
-from tempfile import mkstemp
-
-# import poppunk package
-from .__init__ import __version__
-
-from .sketchlib import removeFromDB
-
-from .utils import storePickle
-from .utils import readPickle
-from .utils import iterDistRows
-
-def prune_distance_matrix(refList, remove_seqs_in, distMat, output):
-    """Rebuild distance matrix following selection of panel of references
-
-    Args:
-        refList (list)
-            List of sequences used to generate distance matrix
-        remove_seqs_in (list)
-            List of sequences to be omitted
-        distMat (numpy.array)
-            nx2 matrix of core distances (column 0) and accessory
-            distances (column 1)
-        output (string)
-            Prefix for new distance output files
-    Returns:
-        newRefList (list)
-            List of sequences retained in distance matrix
-        newDistMat (numpy.array)
-            Updated version of distMat
-    """
-    # Find list items to remove
-    remove_seqs_list = []
-    removal_indices = []
-    for to_remove in remove_seqs_in:
-        found = False
-        for idx, item in enumerate(refList):
-            if item == to_remove:
-                removal_indices.append(idx)
-                remove_seqs_list.append(item)
-                found = True
-                break
-        if not found:
-            sys.stderr.write("Couldn't find " + to_remove + " in database\n")
-    remove_seqs = frozenset(remove_seqs_list)
-
-    if len(remove_seqs) > 0:
-        sys.stderr.write("Removing " + str(len(remove_seqs)) + " sequences\n")
-
-        numNew = len(refList) - len(remove_seqs)
-        newDistMat = np.zeros((int(0.5 * numNew * (numNew - 1)), 2), dtype=distMat.dtype)
-
-        # Create new reference list iterator
-        removal_indices.sort()
-        removal_indices.reverse()
-        next_remove = removal_indices.pop()
-        newRefList = []
-        for idx, seq in enumerate(refList):
-            if idx == next_remove:
-                if len(removal_indices) > 0:
-                    next_remove = removal_indices.pop()
-            else:
-                newRefList.append(seq)
-
-        newRowNames = iter(iterDistRows(newRefList, newRefList, self=True))
-
-        # Copy over rows which don't have an excluded sequence
-        newIdx = 0
-        for distRow, (ref1, ref2) in zip(distMat, iterDistRows(refList, refList, self=True)):
-            if ref1 not in remove_seqs and ref2 not in remove_seqs:
-                (newRef1, newRef2) = next(newRowNames)
-                if newRef1 == ref1 and newRef2 == ref2:
-                    newDistMat[newIdx, :] = distRow
-                    newIdx += 1
-                else:
-                    raise RuntimeError("Row name mismatch. Old: " + ref1 + "," + ref2 + "\n"
-                                       "New: " + newRef1 + "," + newRef2 + "\n")
-
-        storePickle(newRefList, newRefList, True, newDistMat, output)
-    else:
-        newRefList = refList
-        newDistMat = distMat
-
-    # return new distance matrix and sequence lists
-    return newRefList, newDistMat
-
-# command line parsing
-def get_options():
-
-    import argparse
-
-    parser = argparse.ArgumentParser(description='Remove sequences from a PopPUNK database',
-                                     prog='poppunk_prune')
-
-    # input options
-    iGroup = parser.add_argument_group('Input files')
-    iGroup.add_argument('--remove', required=True, help='File listing sequences to remove (required)')
-    iGroup.add_argument('--distances', required=True, help='Prefix of input pickle of pre-calculated distances (required)')
-    iGroup.add_argument('--ref-db', help='Location of reference db, if resketching')
-
-    # output options
-    oGroup = parser.add_argument_group('Output options')
-    oGroup.add_argument('--output', required=True, help='Prefix for output files (required)')
-
-    # processing
-    other = parser.add_argument_group('Other options')
-    other.add_argument('--threads', default=1, type=int, help='Number of threads to use [default = 1]')
-
-    other.add_argument('--version', action='version',
-                       version='%(prog)s '+__version__)
-
-    return parser.parse_args()
-
-def main():
-
-    # Check input ok
-    args = get_options()
-
-    # Create the output directory
-    if not os.path.isdir(args.output):
-        try:
-            os.makedirs(args.output)
-        except OSError:
-            sys.stderr.write("Cannot create output directory\n")
-            sys.exit(1)
-
-    # Read in old distances
-    refList, queryList, self, distMat = readPickle(args.distances,
-                                                   enforce_self=True)
-
-    # Read in seqs to remove
-    remove_seqs_in = []
-    with open(args.remove, 'r') as remove_file:
-        for line in remove_file:
-            remove_seqs_in.append(line.rstrip())
-
-    # reduce distance matrix
-    newRefList = prune_distance_matrix(refList,
-                                       remove_seqs_in,
-                                       distMat,
-                                       args.output + "/" + os.path.basename(args.output))[0]
-
-    if len(refList) != len(newRefList):
-        if args.ref_db is not None:
-            sys.stderr.write("Resketching sequences\n")
-            removeFromDB(args.ref_db, args.output, remove_seqs_in)
-            os.rename(args.output + "/" + os.path.basename(args.output) + ".tmp.h5",
-                      args.output + "/" + os.path.basename(args.output) + ".h5")
-    else:
-        sys.stderr.write("No sequences to remove\n")
-
-    sys.exit(0)
-
diff --git a/PopPUNK/qc.py b/PopPUNK/qc.py
new file mode 100755
index 00000000..14e822d6
--- /dev/null
+++ b/PopPUNK/qc.py
@@ -0,0 +1,529 @@
+#!/usr/bin/env python
+# vim: set fileencoding=<utf-8> :
+# Copyright 2018-2022 John Lees and Nick Croucher
+
+# universal
+import os
+import sys
+# additional
+import numpy as np
+from collections import Counter
+
+import poppunk_refine
+
+from .utils import storePickle, iterDistRows, readIsolateTypeFromCsv
+
+def prune_distance_matrix(refList, remove_seqs_in, distMat, output):
+    """Rebuild distance matrix following selection of panel of references
+
+    Args:
+        refList (list)
+            List of sequences used to generate distance matrix
+        remove_seqs_in (list)
+            List of sequences to be omitted
+        distMat (numpy.array)
+            nx2 matrix of core distances (column 0) and accessory
+            distances (column 1)
+        output (string)
+            Prefix for new distance output files
+    Returns:
+        newRefList (list)
+            List of sequences retained in distance matrix
+        newDistMat (numpy.array)
+            Updated version of distMat
+    """
+    # Find list items to remove
+    remove_seqs_list = []
+    removal_indices = []
+    for to_remove in remove_seqs_in:
+        found = False
+        for idx, item in enumerate(refList):
+            if item == to_remove:
+                removal_indices.append(idx)
+                remove_seqs_list.append(item)
+                found = True
+                break
+        if not found:
+            sys.stderr.write("Couldn't find " + to_remove + " in database\n")
+    remove_seqs = frozenset(remove_seqs_list)
+
+    if len(remove_seqs) > 0:
+        sys.stderr.write("Removing " + str(len(remove_seqs)) + " sequences\n")
+
+        numNew = len(refList) - len(remove_seqs)
+        newDistMat = np.zeros((int(0.5 * numNew * (numNew - 1)), 2), dtype=distMat.dtype)
+
+        # Create new reference list iterator
+        removal_indices.sort()
+        removal_indices.reverse()
+        next_remove = removal_indices.pop()
+        newRefList = []
+        for idx, seq in enumerate(refList):
+            if idx == next_remove:
+                if len(removal_indices) > 0:
+                    next_remove = removal_indices.pop()
+            else:
+                newRefList.append(seq)
+
+        newRowNames = iter(iterDistRows(newRefList, newRefList, self=True))
+
+        # Copy over rows which don't have an excluded sequence
+        # TODO this seems like it would be slow for a big dist matrix
+        # TODO didn't we used to have this as a slice?
+        newIdx = 0
+        for distRow, (ref1, ref2) in zip(distMat, iterDistRows(refList, refList, self=True)):
+            if ref1 not in remove_seqs and ref2 not in remove_seqs:
+                (newRef1, newRef2) = next(newRowNames)
+                if newRef1 == ref1 and newRef2 == ref2:
+                    newDistMat[newIdx, :] = distRow
+                    newIdx += 1
+                else:
+                    raise RuntimeError("Row name mismatch. Old: " + ref1 + "," + ref2 + "\n"
+                                       "New: " + newRef1 + "," + newRef2 + "\n")
+
+        storePickle(newRefList, newRefList, True, newDistMat, output)
+    else:
+        newRefList = refList
+        newDistMat = distMat
+
+    # return new distance matrix and sequence lists
+    return newRefList, newDistMat
+
+def prune_query_distance_matrix(refList, queryList, remove_seqs, qrDistMat,
+                                queryAssign=None):
+    """Remove chunks from the distance matrix which correspond to bad queries
+
+    Args:
+        refList (list)
+            List of ref sequences used to generate distance matrix
+        queryList (list)
+            List of query sequences used to generate distance matrix
+        remove_seqs_in (list)
+            List of sequences to be omitted (must be from queries)
+        qrDistMat (numpy.array)
+            (r*q)x2 matrix of core distances (column 0) and accessory
+            distances (column 1)
+        queryAssign (numpy.array)
+            Query assign results, which will also be pruned if passed
+            [default = None]
+    Returns:
+        passing_queries (list)
+            List of query sequences retained in distance matrix
+        newqrDistMat (numpy.array)
+            Updated version of qrDistMat
+        queryAssign (numpy.array)
+            Updated version of queryAssign
+    """
+    if remove_seqs.intersection(refList):
+        raise RuntimeError("Trying to remove references")
+
+    passing_queries = []
+    pass_rows = []
+    for name in queryList:
+        if name not in remove_seqs:
+            passing_queries.append(name)
+            pass_rows += [True] * len(refList)
+        else:
+            pass_rows += [False] * len(refList)
+
+    qrDistMat = qrDistMat[pass_rows, :]
+    if queryAssign is not None:
+        queryAssign = queryAssign[pass_rows]
+
+    return passing_queries, qrDistMat, queryAssign
+
+def sketchlibAssemblyQC(prefix, names, qc_dict):
+    """Calculates random match probability based on means of genomes
+    in assemblyList, and looks for length outliers.
+
+    Args:
+        prefix (str)
+            Prefix of output files
+        names (list)
+            Names of samples to QC
+        qc_dict (dict)
+            Dictionary of QC parameters
+
+    Returns:
+        retained (list)
+            List of sequences passing QC filters
+        failed (dict)
+            List of sequences failing, and reasons
+    """
+    import h5py
+    from .sketchlib import removeFromDB
+
+    sys.stderr.write("Running QC on sketches\n")
+
+    # open databases
+    db_name = prefix + '/' + os.path.basename(prefix) + '.h5'
+    hdf_in = h5py.File(db_name, 'r')
+
+    # try/except structure to prevent h5 corruption
+    failed_samples = {}
+    try:
+        # process data structures
+        read_grp = hdf_in['sketches']
+
+        seq_length = {}
+        seq_ambiguous = {}
+        # iterate through sketches
+        for dataset in read_grp:
+            if dataset in names:
+                # test thresholds
+                seq_length[dataset] = hdf_in['sketches'][dataset].attrs['length']
+                # If reads, do not QC based on Ns (simpler this way)
+                # Older versions of DB do not save reads, so attr may not be present
+                if 'reads' in hdf_in['sketches'][dataset].attrs and \
+                    hdf_in['sketches'][dataset].attrs['reads']:
+                    seq_ambiguous[dataset] = 0
+                else:
+                    seq_ambiguous[dataset] = hdf_in['sketches'][dataset].attrs['missing_bases']
+
+        # calculate thresholds
+        # get mean length
+        genome_lengths = np.fromiter(seq_length.values(), dtype = int)
+        mean_genome_length = np.mean(genome_lengths)
+
+        # calculate length threshold unless user-supplied
+        if qc_dict['length_range'][0] is None:
+            lower_length = mean_genome_length - \
+                qc_dict['length_sigma'] * np.std(genome_lengths)
+            upper_length = mean_genome_length + \
+                qc_dict['length_sigma'] * np.std(genome_lengths)
+        else:
+            lower_length, upper_length = qc_dict['length_range']
+
+        # open file to report QC failures
+        for dataset in seq_length.keys():
+            # determine if sequence passes filters
+            if seq_length[dataset] < lower_length:
+                failed_samples[dataset] = ['Below lower length threshold']
+            elif seq_length[dataset] > upper_length:
+                failed_samples[dataset] = ['Above upper length threshold']
+            if (qc_dict['upper_n'] is not None and seq_ambiguous[dataset] > qc_dict['upper_n']) or \
+                (seq_ambiguous[dataset] > qc_dict['prop_n'] * seq_length[dataset]):
+                message = "Ambiguous sequence too high"
+                if dataset in failed_samples:
+                    failed_samples[dataset].append(message)
+                else:
+                    failed_samples[dataset] = [message]
+
+        hdf_in.close()
+    # if failure still close files to avoid corruption
+    except:
+        hdf_in.close()
+        sys.stderr.write('Problem processing h5 databases during QC - aborting\n')
+
+        print("Unexpected error:", sys.exc_info()[0], file = sys.stderr)
+        raise
+
+    # This gives back retained in the same order as names
+    retained_samples = [x for x in names if x not in frozenset(failed_samples.keys())]
+
+    return retained_samples, failed_samples
+
+
+def qcDistMat(distMat, refList, queryList, ref_db, qc_dict):
+    """Checks distance matrix for outliers.
+
+    Args:
+        distMat (np.array)
+            Core and accessory distances
+        refList (list)
+            Reference labels
+        queryList (list)
+            Query labels (or refList if self)
+        ref_db (str)
+            Prefix of reference database
+        qc_dict (dict)
+            Dict of QC options
+
+    Returns:
+        retained (list)
+            List of sequences passing QC filters
+        failed (dict)
+            List of sequences failing, and reasons
+    """
+    # Create overall list of sequences
+    if refList == queryList:
+        names = refList
+        self = True
+    else:
+        names = refList + queryList
+        self = False
+
+    # Pick type isolate if not supplied
+    if qc_dict['type_isolate'] is None:
+        qc_dict['type_isolate'] = pickTypeIsolate(ref_db, refList)
+        sys.stderr.write('Selected type isolate for distance QC is ' + qc_dict['type_isolate'] + '\n')
+
+    # First check with numpy, which is quicker than iterating over everything
+    #long_distance_rows = np.where([(distMat[:, 0] > qc_dict['max_pi_dist']) | (distMat[:, 1] > qc_dict['max_a_dist'])])[1].tolist()
+    long_distance_rows = np.where([(distMat[:, 0] > qc_dict['max_pi_dist']) | (distMat[:, 1] > qc_dict['max_a_dist'])],0,1)[0].tolist()
+    long_edges = poppunk_refine.generateTuples(long_distance_rows,
+                                                0,
+                                                self = self,
+                                                num_ref = len(refList),
+                                                int_offset = 0)
+
+    failed = prune_edges(long_edges,
+                                 type_isolate=names.index(qc_dict['type_isolate']),
+                                 query_start=len(refList),
+                                 allow_ref_ref=self)
+    # Convert the edge IDs back to sample names
+    failed_samples = {names[x]: ["Failed distance QC (too high)"] for x in failed}
+
+    # Check if too many zeros, basically the same way but update the existing
+    # dicts/sets, and set a minimum count of zero lengths
+    if qc_dict["prop_zero"] < 1:
+        zero_count = round(qc_dict["prop_zero"] * len(names))
+        zero_distance_rows = np.where([(distMat[:, 0] == 0) | (distMat[:, 1] == 0)],0,1)[0].tolist()
+        zero_edges = poppunk_refine.generateTuples(zero_distance_rows,
+                                                    0,
+                                                    self = self,
+                                                    num_ref = len(refList),
+                                                    int_offset = 0)
+        failed = prune_edges(zero_edges,
+                            type_isolate=names.index(qc_dict['type_isolate']),
+                            query_start=len(refList),
+                            failed=failed,
+                            min_count=zero_count,
+                            allow_ref_ref=self)
+        message = ["Failed distance QC (too many zeros)"]
+        for sample in failed:
+            name = names[sample]
+            if name in failed_samples:
+                failed_samples[name] += message
+            else:
+                failed_samples[name] = message
+
+    retained_samples = [x for x in names if x not in frozenset(failed_samples.keys())]
+    return retained_samples, failed_samples
+
+
+def qcQueryAssignments(rList, qList, query_assignments, max_clusters,
+                       original_cluster_file):
+    """Checks assignments for too many links between clusters.
+
+    Args:
+        refList (list)
+            Reference labels
+        queryList (list)
+            Query labels
+        query_assignments (list or np.array)
+            List of assignments for qrMat, where -1 is a link
+        max_clusters (int)
+            Maximum number of clusters which can be connected
+        original_cluster_file (str)
+            File to load original cluster definitions from
+
+    Returns:
+        retained (list)
+            List of sequences passing QC filters
+        failed (dict)
+            List of sequences failing, and reasons
+    """
+    message = ["Failed graph QC (too many links)"]
+    retained_samples = []
+    failed_samples = {}
+
+    # Read the rList cluster assignments, and turn into a dict which
+    # is idx: cluster
+    clusters = readIsolateTypeFromCsv(original_cluster_file, return_dict=True)
+    clusters_idx = {idx: clusters['Cluster'][name] for idx, name in enumerate(rList)}
+
+    # Find the edges for each query, and count how many unique clusters they
+    # appear in
+    for idx, query in enumerate(qList):
+        row_start = idx * len(rList)
+        row_end = (idx + 1) * len(rList)
+        edges = np.argwhere(query_assignments[row_start:row_end] == -1).reshape(-1)
+        cluster_links = set()
+        for edge in edges:
+            cluster_links.add(clusters_idx[edge])
+
+        if len(cluster_links) > max_clusters:
+            failed_samples[query] = message
+        else:
+            retained_samples.append(query)
+    return retained_samples, failed_samples
+
+def prune_edges(long_edges, type_isolate, query_start,
+                failed=None, min_count=1, allow_ref_ref=True):
+    """Gives a list of failed vertices from a list of failing
+    edges. Tries to prune by those nodes with highest degree of
+    bad nodes, preferentially removes queries, and doesn't remove
+    the type isolate
+
+    Args:
+        long_edges (list of tuples)
+            List of bad edges as node IDs
+        type_isolate (int)
+            The node ID of the type isolate
+        query_start (int)
+            The first node ID which corresponds to queries
+        failed (set or None)
+            If set, an existing prune list to add to
+        min_count (int)
+            Must be at least this many failures to prune
+        allow_ref_ref (bool)
+            Whether r-r edges can be pruned (set False if querying)
+    Returns:
+        failed (set)
+            Failed sample IDs
+    """
+    if failed == None:
+        failed = set()
+    if len(long_edges) > 0:
+        # Find nodes with the most bad edges
+        counts = Counter()
+        for (r, q) in long_edges:
+            counts.update([r, q])
+
+        # Sorts by edges which appear most often
+        long_edges.sort(key=lambda x: max(counts[x[0]], counts[x[1]]), reverse=True)
+        for (r, q) in long_edges:
+            if q not in failed and r not in failed and (counts[r] >= min_count or counts[q] >= min_count):
+                # Do not add any refs if querying
+                if r < query_start and q < query_start:
+                    if allow_ref_ref:
+                        if q == type_isolate or (counts[r] > counts[q] and counts[r] >= min_count):
+                            failed.add(r)
+                        elif counts[q] >= min_count:
+                            failed.add(q)
+                # NB q > r
+                elif r < query_start and q >= query_start:
+                    failed.add(q)
+                else:
+                    if counts[r] > counts[q] and counts[r] >= min_count:
+                        failed.add(r)
+                    elif counts[q] >= min_count:
+                        failed.add(q)
+
+    return failed
+
+def remove_qc_fail(qc_dict, names, passed, fail_dicts, ref_db, distMat, prefix,
+                   strand_preserved=False, threads=1):
+    """Removes samples failing QC from the database and distances. Also
+    recalculates random match chances.
+
+    Args:
+        qc_dict (dict)
+            Dictionary of QC options
+        names (list)
+            The names of samples in the distMat
+        passed (list)
+            The names of passing samples
+        fail_dicts (list)
+            A list of dictionaries, which each have keys of samples,
+            and values with lists of reasons
+        ref_db (str)
+            Prefix for the database
+        distMat (numpy.array)
+            The distance matrix to prune
+        prefix (str)
+            The prefix to save the new db and distMat to
+        strand_preserved (bool)
+            Whether to use a preserved strand when recalculating random match
+            chances [default = False].
+        threads (int)
+            Number of CPU threads to use when recalculating random match chances
+            [default = 1].
+    """
+    from .sketchlib import removeFromDB, addRandom, readDBParams
+
+    # Create output directory if it does not exist already
+    if not os.path.isdir(prefix):
+        try:
+            os.makedirs(prefix)
+        except OSError:
+            sys.stderr.write("Cannot create output directory " + prefix + "\n")
+            sys.exit(1)
+
+    failed = set(names) - set(passed)
+    if qc_dict['retain_failures']:
+        removeFromDB(ref_db,
+                    f"{prefix}/failed.{os.path.basename(prefix)}.h5",
+                    passed,
+                    full_names = True)
+    # new database file if pruning
+    if failed and not qc_dict['no_remove']:
+        # sys.stderr.write(f"Removing {len(failed)} samples from database and distances\n")
+        tmp_filtered_db_name = f"{prefix}/filtered.{os.path.basename(prefix)}.h5"
+        output_db_name = f"{prefix}/{os.path.basename(prefix)}.h5"
+        input_db_name = f"{ref_db}/{os.path.basename(ref_db)}.h5"
+        removeFromDB(input_db_name,
+                     tmp_filtered_db_name,
+                     failed,
+                     full_names = True)
+        os.rename(tmp_filtered_db_name, output_db_name)
+
+        # Remove from the distMat too
+        prune_distance_matrix(names,
+                              failed,
+                              distMat,
+                              f"{prefix}/{os.path.basename(prefix)}.dists")
+
+        #if any removed, recalculate random
+        sys.stderr.write(f"Recalculating random matches with strand_preserved = {strand_preserved}\n")
+        db_kmers = readDBParams(ref_db)[0]
+        addRandom(prefix, passed, db_kmers, strand_preserved,
+                    overwrite=True, threads=threads)
+
+    # write failing & reasons
+    with open(f"{prefix}/{os.path.basename(prefix)}_qcreport.txt", 'w') as qc_file:
+        for sample in failed:
+            reasons = []
+            for fail_test in fail_dicts:
+                if sample in fail_test:
+                    reasons += (fail_test[sample])
+            qc_file.write(f"{sample}\t{','.join(reasons)}\n")
+
+def pickTypeIsolate(prefix, refList):
+    """Selects a type isolate as that with a minimal proportion
+    of missing data.
+
+    Args:
+        prefix (str)
+            Prefix for database
+        refList (list)
+            References to pick from
+
+    Returns:
+        type_isolate (str)
+            Name of isolate selected as reference
+    """
+    # open databases
+    import h5py
+    db_name = prefix + '/' + os.path.basename(prefix) + '.h5'
+    hdf_in = h5py.File(db_name, 'r')
+
+    min_prop_n = 1.0
+    type_isolate = None
+
+    try:
+        # process data structures
+        read_grp = hdf_in['sketches']
+        # iterate through sketches
+        for sample in refList:
+            if sample in read_grp:
+                if 'reads' in hdf_in['sketches'][sample].attrs and \
+                    hdf_in['sketches'][sample].attrs['reads']:
+                    sample_prop_n = 1.0
+                else:
+                    sample_prop_n = hdf_in['sketches'][sample].attrs['missing_bases']/hdf_in['sketches'][sample].attrs['length']
+                if sample_prop_n < min_prop_n:
+                    min_prop_n = sample_prop_n
+                    type_isolate = sample
+                if min_prop_n == 0.0:
+                    break
+    # if failure still close files to avoid corruption
+    except:
+        hdf_in.close()
+        sys.stderr.write('Problem processing h5 databases during QC - aborting\n')
+        print("Unexpected error:", sys.exc_info()[0], file = sys.stderr)
+        raise
+
+    return type_isolate
+
diff --git a/PopPUNK/reference_pick.py b/PopPUNK/reference_pick.py
index d8b2be5f..472b201d 100755
--- a/PopPUNK/reference_pick.py
+++ b/PopPUNK/reference_pick.py
@@ -17,7 +17,7 @@
 from .network import load_network_file
 from .network import save_network
 
-from .prune_db import prune_distance_matrix
+from .qc import prune_distance_matrix
 
 from .utils import setGtThreads
 from .utils import readPickle
diff --git a/PopPUNK/refine.py b/PopPUNK/refine.py
index 516ff690..50e13870 100644
--- a/PopPUNK/refine.py
+++ b/PopPUNK/refine.py
@@ -12,6 +12,7 @@
 import numpy as np
 import scipy.optimize
 import collections
+from math import sqrt
 from tqdm import tqdm
 try:
     from multiprocessing import Pool, shared_memory
@@ -186,23 +187,10 @@ def refineFit(distMat, sample_names, mean0, mean1, scale,
         search_length = max_move + ((mean1[0] - mean0[0])**2 + (mean1[1] - mean0[1])**2)**0.5
         global_grid_resolution = 40 # Seems to work
         s_range = np.linspace(-min_move, search_length, num = global_grid_resolution)
-        bottom_end = transformLine(s_range[0], mean0, mean1)
-        top_end = transformLine(s_range[-1], mean0, mean1)
-        min_x, min_y = decisionBoundary(bottom_end, gradient)
-        max_x, max_y = decisionBoundary(top_end, gradient)
-
+        (min_x, max_x), (min_y, max_y) = \
+            check_search_range(scale, mean0, mean1, s_range[0], s_range[-1])
         if min_x < 0 or min_y < 0:
             raise RuntimeError("Boundary range below zero")
-        sys.stderr.write("Search range (" +
-                         ",".join(["{:.3f}".format(x) for x in bottom_end * scale]) +
-                         ") to (" +
-                         ",".join(["{:.3f}".format(x) for x in top_end * scale]) + ")\n")
-        sys.stderr.write("Searching core intercept from " +
-                         "{:.3f}".format(min_x * scale[0]) +
-                         " to " + "{:.3f}".format(max_x * scale[0]) + "\n")
-        sys.stderr.write("Searching accessory intercept from " +
-                         "{:.3f}".format(min_y * scale[1]) +
-                         " to " + "{:.3f}".format(max_y * scale[1]) + "\n")
 
         i_vec, j_vec, idx_vec = \
             poppunk_refine.thresholdIterate1D(distMat, s_range, slope,
@@ -245,16 +233,17 @@ def refineFit(distMat, sample_names, mean0, mean1, scale,
         optimised_coor = transformLine(optimised_s, mean0, mean1)
         if slope == 2:
             optimal_x, optimal_y = decisionBoundary(optimised_coor, gradient)
+            if optimal_x < 0 or optimal_y < 0:
+                raise RuntimeError("Optimisation failed: produced a boundary outside of allowed range\n")
         else:
             optimal_x = optimised_coor[0]
             optimal_y = optimised_coor[1]
-
-    if optimal_x < 0 or optimal_y < 0:
-        raise RuntimeError("Optimisation failed: produced a boundary outside of allowed range\n")
+            if (slope == 0 and optimal_x < 0) or (slope == 1 and optimal_y < 0):
+               raise RuntimeError("Optimisation failed: produced a boundary outside of allowed range\n")
 
     return optimal_x, optimal_y, optimised_s
 
-def multi_refine(distMat, sample_names, mean0, mean1, s_max,
+def multi_refine(distMat, sample_names, mean0, mean1, scale, s_max,
                  n_boundary_points, output_prefix,
                  num_processes = 1, use_gpu = False):
     """Move the refinement boundary between the optimum and where it meets an
@@ -269,6 +258,8 @@ def multi_refine(distMat, sample_names, mean0, mean1, s_max,
             Start point to define search line
         mean1 (numpy.array)
             End point to define search line
+        scale (numpy.array)
+            Scaling factor of distMat
         s_max (float)
             The optimal s position from refinement (:func:`~PopPUNK.refine.refineFit`)
         n_boundary_points (int)
@@ -285,13 +276,16 @@ def multi_refine(distMat, sample_names, mean0, mean1, s_max,
     # Set the range
     # Between optimised s and where line meets an axis
     gradient = (mean1[1] - mean0[1]) / (mean1[0] - mean0[0])
-    # Equation of normal passing through origin y = -1/m * x
-    # Where this meets line y - y1 = m(x - x1) is at:
-    x = (gradient * mean0[0] - mean0[1]) / (gradient + 1 / gradient)
-    y = mean0[1] + gradient * (x - mean0[0])
+    if mean0[1] >= gradient * mean0[0]:
+        s_min = -mean0[0] * sqrt(1 + gradient * gradient)
+    else:
+        s_min = -mean0[1] * sqrt(1 + 1 / (gradient * gradient))
+    s_range = np.linspace(s_min, s_max, num = n_boundary_points)
 
-    s_min = -((mean0[0] - x)**2 + (mean0[1] - y)**2)**0.5
-    s_range = np.linspace(s_min, s_max, num = n_boundary_points)[1:]
+    (min_x, max_x), (min_y, max_y) = \
+        check_search_range(scale, mean0, mean1, s_range[0], s_range[-1])
+    if min_x < 0 or min_y < 0:
+        sys.stderr.write("Boundary range below zero")
 
     i_vec, j_vec, idx_vec = \
         poppunk_refine.thresholdIterate1D(distMat, s_range, 2,
@@ -308,6 +302,45 @@ def multi_refine(distMat, sample_names, mean0, mean1, s_max,
                 write_clusters = output_prefix,
                 use_gpu = use_gpu)
 
+def check_search_range(scale, mean0, mean1, lower_s, upper_s):
+    """Checks a search range is within a valid range
+
+    Args:
+        scale (np.array)
+            Rescaling factor to [0, 1] for each axis
+        mean0 (np.array)
+            (x, y) of starting point defining line
+        mean1 (np.array)
+            (x, y) of end point defining line
+        lower_s (float)
+            distance along line to start search
+        upper_s (float)
+            distance along line to end search
+
+    Returns:
+        min_x, max_x
+            minimum and maximum x-intercepts of the search range
+        min_y, max_y
+            minimum and maximum x-intercepts of the search range
+    """
+    gradient = (mean1[1] - mean0[1]) / (mean1[0] - mean0[0])
+    bottom_end = transformLine(lower_s, mean0, mean1)
+    top_end = transformLine(upper_s, mean0, mean1)
+    min_x, min_y = decisionBoundary(bottom_end, gradient)
+    max_x, max_y = decisionBoundary(top_end, gradient)
+
+    sys.stderr.write("Search range (" +
+                        ",".join(["{:.3f}".format(x) for x in bottom_end * scale]) +
+                        ") to (" +
+                        ",".join(["{:.3f}".format(x) for x in top_end * scale]) + ")\n")
+    sys.stderr.write("Searching core intercept from " +
+                        "{:.3f}".format(min_x * scale[0]) +
+                        " to " + "{:.3f}".format(max_x * scale[0]) + "\n")
+    sys.stderr.write("Searching accessory intercept from " +
+                        "{:.3f}".format(min_y * scale[1]) +
+                        " to " + "{:.3f}".format(max_y * scale[1]) + "\n")
+
+    return((min_x, max_x), (min_y, max_y))
 
 def expand_cugraph_network(G, G_extra_df):
     """Reconstruct a cugraph network with additional edges.
@@ -420,9 +453,8 @@ def growNetwork(sample_names, i_vec, j_vec, idx_vec, s_range, score_idx = 0,
                 # Write the cluster output as long as there is at least one
                 # non-trivial cluster
                 if write_clusters and G_summary[0][0] < len(sample_names):
-                    o_prefix = write_clusters + "/" + \
-                        os.path.basename(write_clusters) + \
-                        "_boundary" + str(s)
+                    o_prefix = \
+                        f"{write_clusters}/{os.path.basename(write_clusters)}_boundary{s + 1}"
                     printClusters(G,
                                   sample_names,
                                   outPrefix=o_prefix,
@@ -575,9 +607,12 @@ def readManualStart(startFile):
             Centre of within-strain distribution
         mean1 (numpy.array)
             Centre of between-strain distribution
+        scaled (bool)
+            True if means are scaled between [0,1]
     """
     mean0 = None
     mean1 = None
+    scaled = True
 
     with open(startFile, 'r') as start:
         for line in start:
@@ -592,6 +627,11 @@ def readManualStart(startFile):
                 for mean_val in value.split(','):
                     mean_read.append(float(mean_val))
                 mean1 = np.array(mean_read)
+            elif param == 'scaled':
+                if value == "False" or value == "false":
+                    scaled = False
+            else:
+                raise RuntimeError("Incorrectly formatted manual start file")
     try:
         if not isinstance(mean0, np.ndarray) or not isinstance(mean1, np.ndarray):
             raise RuntimeError('Must set both start and end')
@@ -606,7 +646,7 @@ def readManualStart(startFile):
         sys.stderr.write(str(e) + "\n")
         sys.exit(1)
 
-    return mean0, mean1
+    return mean0, mean1, scaled
 
 def likelihoodBoundary(s, model, start, end, within, between):
     """Wrapper function around :func:`~PopPUNK.bgmm.fit2dMultiGaussian` so that it can
diff --git a/PopPUNK/sketchlib.py b/PopPUNK/sketchlib.py
index 7371343c..839c741f 100644
--- a/PopPUNK/sketchlib.py
+++ b/PopPUNK/sketchlib.py
@@ -8,14 +8,7 @@
 import sys
 import subprocess
 # additional
-import collections
-import pickle
-import time
-from tempfile import mkstemp
-from multiprocessing import Pool, Lock
-from functools import partial
-from itertools import product
-from glob import glob
+import re
 from random import sample
 import numpy as np
 from scipy import optimize
@@ -24,11 +17,10 @@
 import h5py
 
 from .__init__ import SKETCHLIB_MAJOR, SKETCHLIB_MINOR, SKETCHLIB_PATCH
-from .utils import iterDistRows
-from .utils import readRfile
+from .utils import readRfile, stderr_redirected
 from .plot import plot_fit
 
-sketchlib_exe = "poppunk_sketch"
+sketchlib_exe = "sketchlib"
 
 def checkSketchlibVersion():
     """Checks that sketchlib can be run, and returns version
@@ -37,18 +29,24 @@ def checkSketchlibVersion():
         version (str)
             Version string
     """
+    sketchlib_version = [0, 0, 0]
     try:
         version = pp_sketchlib.version
 
     # Older versions didn't export attributes
     except AttributeError:
-        p = subprocess.Popen([sketchlib_exe + ' --version'], shell=True, stdout=subprocess.PIPE)
-        version = 0
-        for line in iter(p.stdout.readline, ''):
-            if line != '':
-                version = line.rstrip().decode().split(" ")[1]
-                break
+        try:
+            p = subprocess.Popen([sketchlib_exe + ' --version'], shell=True, stdout=subprocess.PIPE)
+            version = 0
+            for line in iter(p.stdout.readline, ''):
+                if line != '':
+                    version = line.rstrip().decode().split(" ")[1]
+                    break
 
+        except IndexError:
+            sys.stderr.write("WARNING: Sketchlib version could not be found\n")
+
+    version = re.sub(r'^v', '', version) # Remove leading v
     sketchlib_version = [int(v) for v in version.split(".")]
     if sketchlib_version[0] < SKETCHLIB_MAJOR or \
         sketchlib_version[0] == SKETCHLIB_MAJOR and sketchlib_version[1] < SKETCHLIB_MINOR or \
@@ -263,11 +261,11 @@ def joinDBs(db1, db2, output, update_random = None):
             hdf_join.close()
             if len(sequence_names) > 2:
                 sys.stderr.write("Updating random match chances\n")
-                pp_sketchlib.addRandom(join_prefix + ".tmp",
-                                       sequence_names,
-                                       kmer_size,
-                                       not strand_preserved,
-                                       threads)
+                pp_sketchlib.addRandom(db_name=join_prefix + ".tmp",
+                                       samples=sequence_names,
+                                       klist=kmer_size,
+                                       use_rc=(not strand_preserved),
+                                       num_threads=threads)
         elif 'random' in hdf1:
             hdf1.copy('random', hdf_join)
 
@@ -341,7 +339,7 @@ def removeFromDB(db_name, out_name, removeSeqs, full_names = False):
 def constructDatabase(assemblyList, klist, sketch_size, oPrefix,
                         threads, overwrite,
                         strand_preserved, min_count,
-                        use_exact, qc_dict, calc_random = True,
+                        use_exact, calc_random = True,
                         codon_phased = False,
                         use_gpu = False, deviceid = 0):
     """Sketch the input assemblies at the requested k-mer lengths
@@ -375,8 +373,6 @@ def constructDatabase(assemblyList, klist, sketch_size, oPrefix,
         use_exact (bool)
             Use exact count of k-mer appearance in reads
             (default = False)
-        qc_dict (dict)
-            Dict containg QC settings
         calc_random (bool)
             Add random match chances to DB (turn off for queries)
         codon_phased (bool)
@@ -390,8 +386,7 @@ def constructDatabase(assemblyList, klist, sketch_size, oPrefix,
             (default = 0)
     Returns:
         names (list)
-            List of names included in the database (some may be pruned due
-            to QC)
+            List of names included in the database (from rfile)
     """
     # read file names
     names, sequences = readRfile(assemblyList)
@@ -404,43 +399,31 @@ def constructDatabase(assemblyList, klist, sketch_size, oPrefix,
         os.remove(dbfilename)
 
     # generate sketches
-    pp_sketchlib.constructDatabase(dbname,
-                                   names,
-                                   sequences,
-                                   klist,
-                                   sketch_size,
-                                   codon_phased,
-                                   False,
-                                   not strand_preserved,
-                                   min_count,
-                                   use_exact,
-                                   threads,
-                                   use_gpu,
-                                   deviceid)
-
-    # QC sequences
-    if qc_dict['run_qc']:
-        filtered_names = sketchlibAssemblyQC(oPrefix,
-                                             names,
-                                             klist,
-                                             qc_dict,
-                                             strand_preserved,
-                                             threads)
-    else:
-        filtered_names = names
+    pp_sketchlib.constructDatabase(db_name=dbname,
+                                   samples=names,
+                                   files=sequences,
+                                   klist=klist,
+                                   sketch_size=sketch_size,
+                                   codon_phased=codon_phased,
+                                   calc_random=False,
+                                   use_rc=not strand_preserved,
+                                   min_count=min_count,
+                                   exact=use_exact,
+                                   num_threads=threads,
+                                   use_gpu=use_gpu,
+                                   device_id=deviceid)
 
     # Add random matches if required
     # (typically on for reference, off for query)
     if (calc_random):
         addRandom(oPrefix,
-                  filtered_names,
+                  names,
                   klist,
                   strand_preserved,
                   overwrite = True,
                   threads = threads)
 
-    # return filtered file names
-    return filtered_names
+    return names
 
 
 def addRandom(oPrefix, sequence_names, klist,
@@ -475,11 +458,11 @@ def addRandom(oPrefix, sequence_names, klist,
                 return
 
         hdf_in.close()
-        pp_sketchlib.addRandom(dbname,
-                               sequence_names,
-                               klist,
-                               not strand_preserved,
-                               threads)
+        pp_sketchlib.addRandom(db_name=dbname,
+                               samples=sequence_names,
+                               klist=klist,
+                               use_rc=(not strand_preserved),
+                               num_threads=threads)
 
 def queryDatabase(rNames, qNames, dbPrefix, queryPrefix, klist, self = True, number_plot_fits = 0,
                   threads = 1, use_gpu = False, deviceid = 0):
@@ -534,8 +517,16 @@ def queryDatabase(rNames, qNames, dbPrefix, queryPrefix, klist, self = True, num
         qNames = rNames
 
         # Calls to library
-        distMat = pp_sketchlib.queryDatabase(ref_db, ref_db, rNames, rNames, klist,
-                                             True, False, threads, use_gpu, deviceid)
+        distMat = pp_sketchlib.queryDatabase(ref_db_name=ref_db,
+                                             query_db_name=ref_db,
+                                             rList=rNames,
+                                             qList=rNames,
+                                             klist=klist,
+                                             random_correct=True,
+                                             jaccard=False,
+                                             num_threads=threads,
+                                             use_gpu=use_gpu,
+                                             device_id=deviceid)
 
         # option to plot core/accessory fits. Choose a random number from cmd line option
         if number_plot_fits > 0:
@@ -544,23 +535,24 @@ def queryDatabase(rNames, qNames, dbPrefix, queryPrefix, klist, self = True, num
                 example = sample(rNames, k=2)
                 raw = np.zeros(len(klist))
                 corrected = np.zeros(len(klist))
-                raw = pp_sketchlib.queryDatabase(ref_db,
-                                                    ref_db,
-                                                    [example[0]],
-                                                    [example[1]],
-                                                    klist,
-                                                    random_correct = False,
-                                                    jaccard = True,
-                                                    num_threads = threads,
+                with stderr_redirected(): # Hide the many progress bars
+                    raw = pp_sketchlib.queryDatabase(ref_db_name=ref_db,
+                                                    query_db_name=ref_db,
+                                                    rList=[example[0]],
+                                                    qList=[example[1]],
+                                                    klist=klist,
+                                                    random_correct=False,
+                                                    jaccard=True,
+                                                    num_threads=threads,
                                                     use_gpu = False)
-                corrected = pp_sketchlib.queryDatabase(ref_db,
-                                                        ref_db,
-                                                        [example[0]],
-                                                        [example[1]],
-                                                        klist,
-                                                        random_correct = True,
-                                                        jaccard = True,
-                                                        num_threads = threads,
+                    corrected = pp_sketchlib.queryDatabase(ref_db_name=ref_db,
+                                                        query_db_name=ref_db,
+                                                        rList=[example[0]],
+                                                        qList=[example[1]],
+                                                        klist=klist,
+                                                        random_correct=True,
+                                                        jaccard=True,
+                                                        num_threads=threads,
                                                         use_gpu = False)
                 raw_fit = fitKmerCurve(raw[0], klist, jacobian)
                 corrected_fit = fitKmerCurve(corrected[0], klist, jacobian)
@@ -581,31 +573,40 @@ def queryDatabase(rNames, qNames, dbPrefix, queryPrefix, klist, self = True, num
 
         # Calls to library
         query_db = queryPrefix + "/" + os.path.basename(queryPrefix)
-        distMat = pp_sketchlib.queryDatabase(ref_db, query_db, rNames, qNames, klist,
-                                             True, False, threads, use_gpu, deviceid)
+        distMat = pp_sketchlib.queryDatabase(ref_db_name=ref_db,
+                                             query_db_name=query_db,
+                                             rList=rNames,
+                                             qList=qNames,
+                                             klist=klist,
+                                             random_correct=True,
+                                             jaccard=False,
+                                             num_threads=threads,
+                                             use_gpu=use_gpu,
+                                             device_id=deviceid)
 
         # option to plot core/accessory fits. Choose a random number from cmd line option
         if number_plot_fits > 0:
             jacobian = -np.hstack((np.ones((klist.shape[0], 1)), klist.reshape(-1, 1)))
             ref_examples = sample(rNames, k = number_plot_fits)
             query_examples = sample(qNames, k = number_plot_fits)
-            raw = pp_sketchlib.queryDatabase(ref_db,
-                                                query_db,
-                                                ref_examples,
-                                                query_examples,
-                                                klist,
-                                                random_correct = False,
-                                                jaccard = True,
-                                                num_threads = threads,
+            with stderr_redirected(): # Hide the many progress bars
+                raw = pp_sketchlib.queryDatabase(ref_db_name=ref_db,
+                                                query_db_name=query_db,
+                                                rList=ref_examples,
+                                                qList=query_examples,
+                                                klist=klist,
+                                                random_correct=False,
+                                                jaccard=True,
+                                                num_threads=threads,
                                                 use_gpu = False)
-            corrected = pp_sketchlib.queryDatabase(ref_db,
-                                                    query_db,
-                                                    ref_examples,
-                                                    query_examples,
-                                                    klist,
-                                                    random_correct = True,
-                                                    jaccard = True,
-                                                    num_threads = threads,
+                corrected = pp_sketchlib.queryDatabase(ref_db_name=ref_db,
+                                                    query_db_name=query_db,
+                                                    rList=ref_examples,
+                                                    qList=query_examples,
+                                                    klist=klist,
+                                                    random_correct=True,
+                                                    jaccard=True,
+                                                    num_threads=threads,
                                                     use_gpu = False)
             for plot_idx in range(number_plot_fits):
                 raw_fit = fitKmerCurve(raw[plot_idx], klist, jacobian)
@@ -622,188 +623,6 @@ def queryDatabase(rNames, qNames, dbPrefix, queryPrefix, klist, self = True, num
     return distMat
 
 
-def pickTypeIsolate(prefix, names):
-    """Selects a type isolate as that with a minimal proportion
-    of missing data.
-
-    Args:
-        prefix (str)
-            Prefix of output files
-        names (list)
-            Names of samples to QC
-
-    Returns:
-        type_isolate (str)
-            Name of isolate selected as reference
-    """
-    # open databases
-    db_name = prefix + '/' + os.path.basename(prefix) + '.h5'
-    hdf_in = h5py.File(db_name, 'r')
-
-    min_prop_n = 1.0
-    type_isolate = None
-
-    try:
-        # process data structures
-        read_grp = hdf_in['sketches']
-        # iterate through sketches
-        for dataset in read_grp:
-            if hdf_in['sketches'][dataset].attrs['missing_bases']/hdf_in['sketches'][dataset].attrs['length'] < min_prop_n:
-                min_prop_n = hdf_in['sketches'][dataset].attrs['missing_bases']/hdf_in['sketches'][dataset].attrs['length']
-                type_isolate = dataset
-            if min_prop_n == 0.0:
-                break
-    # if failure still close files to avoid corruption
-    except:
-        hdf_in.close()
-        sys.stderr.write('Problem processing h5 databases during QC - aborting\n')
-        print("Unexpected error:", sys.exc_info()[0], file = sys.stderr)
-        raise
-
-    return type_isolate
-
-def sketchlibAssemblyQC(prefix, names, klist, qc_dict, strand_preserved, threads):
-    """Calculates random match probability based on means of genomes
-    in assemblyList, and looks for length outliers.
-
-    Args:
-        prefix (str)
-            Prefix of output files
-        names (list)
-            Names of samples to QC
-        klist (list)
-            List of k-mer sizes to sketch
-        qc_dict (dict)
-            Dictionary of QC parameters
-        strand_preserved (bool)
-            Ignore reverse complement k-mers (default = False)
-        threads (int)
-            Number of threads to use in parallelisation
-
-    Returns:
-        retained (list)
-            List of sequences passing QC filters
-    """
-    sys.stderr.write("Running QC on sketches\n")
-
-    # open databases
-    db_name = prefix + '/' + os.path.basename(prefix) + '.h5'
-    hdf_in = h5py.File(db_name, 'r')
-
-    # try/except structure to prevent h5 corruption
-    failed_samples = False
-    try:
-        # process data structures
-        read_grp = hdf_in['sketches']
-
-        seq_length = {}
-        seq_ambiguous = {}
-        retained = []
-        failed = []
-
-        # iterate through sketches
-        for dataset in read_grp:
-            if dataset in names:
-                # test thresholds
-                remove = False
-                seq_length[dataset] = hdf_in['sketches'][dataset].attrs['length']
-                seq_ambiguous[dataset] = hdf_in['sketches'][dataset].attrs['missing_bases']
-
-        # calculate thresholds
-        # get mean length
-        genome_lengths = np.fromiter(seq_length.values(), dtype = int)
-        mean_genome_length = np.mean(genome_lengths)
-
-        # calculate length threshold unless user-supplied
-        if qc_dict['length_range'][0] is None:
-            lower_length = mean_genome_length - \
-                qc_dict['length_sigma'] * np.std(genome_lengths)
-            upper_length = mean_genome_length + \
-                qc_dict['length_sigma'] * np.std(genome_lengths)
-        else:
-            lower_length, upper_length = qc_dict['length_range']
-
-        # open file to report QC failures
-        with open(prefix + '/' + os.path.basename(prefix) + '_qcreport.txt', 'a+') as qc_file:
-            # iterate through and filter
-            for dataset in seq_length.keys():
-                # determine if sequence passes filters
-                remove = False
-                if seq_length[dataset] < lower_length:
-                    remove = True
-                    qc_file.write(dataset + '\tBelow lower length threshold\n')
-                elif seq_length[dataset] > upper_length:
-                    remove = True
-                    qc_file.write(dataset + '\tAbove upper length threshold\n')
-                if qc_dict['upper_n'] is not None and seq_ambiguous[dataset] > qc_dict['upper_n']:
-                    remove = True
-                    qc_file.write(dataset + '\tAmbiguous sequence too high\n')
-                elif seq_ambiguous[dataset] > qc_dict['prop_n'] * seq_length[dataset]:
-                    remove = True
-                    qc_file.write(dataset + '\tAmbiguous sequence too high\n')
-
-                if remove:
-                    sys.stderr.write(dataset + ' failed QC\n')
-                    failed_samples = True
-                    failed.append(dataset)
-                else:
-                    retained.append(dataset)
-
-            # retain sketches of failed samples
-            if qc_dict['retain_failures']:
-                removeFromDB(db_name,
-                             prefix + '/' + 'failed.' + os.path.basename(prefix) + '.h5',
-                             retained,
-                             full_names = True)
-            # new database file if pruning
-            if qc_dict['qc_filter'] == 'prune':
-                filtered_db_name = prefix + '/' + 'filtered.' + os.path.basename(prefix) + '.h5'
-                removeFromDB(db_name,
-                             prefix + '/' + 'filtered.' + os.path.basename(prefix) + '.h5',
-                             failed,
-                             full_names = True)
-                os.rename(filtered_db_name, db_name)
-        hdf_in.close()
-    # if failure still close files to avoid corruption
-    except:
-        hdf_in.close()
-        sys.stderr.write('Problem processing h5 databases during QC - aborting\n')
-
-        print("Unexpected error:", sys.exc_info()[0], file = sys.stderr)
-        raise
-
-    # stop if at least one sample fails QC and option is not continue/prune
-    if failed_samples and qc_dict['qc_filter'] == 'stop':
-        sys.stderr.write('Sequences failed QC filters - details in ' + \
-                         prefix + '/' + os.path.basename(prefix) + \
-                         '_qcreport.txt\n')
-        sys.exit(1)
-    elif qc_dict['qc_filter'] == 'continue':
-        retained = retained + failed
-
-    # stop if no sequences pass QC
-    if len(retained) == 0:
-        sys.stderr.write('No sequences passed QC filters - please adjust your settings\n')
-        sys.exit(1)
-
-    # remove random matches if already present
-    if 'random' in hdf_in:
-        hdf_in.close()
-        hdf_in = h5py.File(db_name, 'r+')
-        del hdf_in['random']
-    hdf_in.close()
-
-    # This gives back retained in the same order as names
-    retained = [x for x in names if x in frozenset(retained)]
-
-    # stop if type sequence does not pass QC or is absent
-    if qc_dict['type_isolate'] is not None and qc_dict['type_isolate'] not in retained:
-        sys.stderr.write('Type isolate ' + qc_dict['type_isolate'] + ' not found in isolates after QC; check '
-        'name of type isolate and QC options\n')
-        sys.exit(1)
-
-    return retained
-
 def fitKmerCurve(pairwise, klist, jacobian):
     """Fit the function :math:`pr = (1-a)(1-c)^k`
 
@@ -836,7 +655,7 @@ def fitKmerCurve(pairwise, klist, jacobian):
                          "\nWith mash input " +
                          np.array2string(pairwise, precision=4, separator=',',suppress_small=True) +
                          "\nCheck for low quality input genomes\n")
-        exit(0)
+        transformed_params = [0, 0]
 
     # Return core, accessory
     return(np.flipud(transformed_params))
diff --git a/PopPUNK/trees.py b/PopPUNK/trees.py
index e240037d..abe22be1 100644
--- a/PopPUNK/trees.py
+++ b/PopPUNK/trees.py
@@ -5,10 +5,15 @@
 
 import sys
 import os
+import io
 import subprocess
 import numpy as np
 import pandas as pd
-import dendropy
+
+from Bio import Phylo
+from Bio.Phylo.TreeConstruction import DistanceTreeConstructor
+from Bio.Phylo.TreeConstruction import DistanceMatrix
+from treeswift import Tree, Node
 
 from .utils import check_and_set_gpu
 
@@ -76,7 +81,7 @@ def buildRapidNJ(rapidnj, refList, coreMat, outPrefix, threads = 1):
         sys.exit(1)
 
     # read tree and return
-    tree = dendropy.Tree.get(path=tree_filename, schema="newick", preserve_underscores=True)
+    tree = Phylo.read(tree_filename, "newick")
     os.remove(tree_filename)
     return tree
 
@@ -100,6 +105,24 @@ def write_tree(tree, prefix, suffix, overwrite):
     else:
         sys.stderr.write("Unable to write phylogeny to " + tree_filename + "\n")
 
+def tree_as_string(tree):
+    """Converts a Phylo object into a newick string
+
+    Args:
+        tree (Phylo)
+            Phylo object
+
+    Returns:
+        tree_string (str)
+            Newick formatted string representing tree
+    """
+    output_str = io.StringIO()
+    Phylo.write(tree, output_str, "newick")
+    output_str.seek(0)
+    tree_string = output_str.getvalue()
+    tree_string = tree_string.replace("'","")
+    return tree_string
+
 def load_tree(prefix, type, distances = 'core'):
     """Checks for existing trees from previous runs.
 
@@ -119,10 +142,8 @@ def load_tree(prefix, type, distances = 'core'):
             tree_fn = tree_prefix + suffix
             if os.path.isfile(tree_fn):
                 sys.stderr.write("Reading existing tree from " + tree_fn + "\n")
-                tree = dendropy.Tree.get(path=tree_fn, schema="newick", preserve_underscores=True)
-                tree_string = tree.as_string(schema="newick",
-                suppress_rooting=True,
-                unquoted_underscores=True)
+                tree = Phylo.read(tree_fn, "newick")
+                tree_string = tree_as_string(tree)
                 break
 
     return tree_string
@@ -150,32 +171,24 @@ def generate_nj_tree(coreMat, seqLabels, outPrefix, rapidnj, threads):
             Newick-formatted string of NJ tree
     """
     # Save distances to file
-    core_dist_file = outPrefix + "/" + os.path.basename(outPrefix) + "_core_dists.csv"
-    np.savetxt(core_dist_file, coreMat, delimiter=",", header = ",".join(seqLabels), comments="")
 
     # calculate phylogeny
     sys.stderr.write("Building phylogeny\n")
     if rapidnj is not None:
+        core_dist_file = outPrefix + "/" + os.path.basename(outPrefix) + "_core_dists.csv"
+        np.savetxt(core_dist_file, coreMat, delimiter=",", header = ",".join(seqLabels), comments="")
         tree = buildRapidNJ(rapidnj, seqLabels, coreMat, outPrefix, threads = threads)
+        os.remove(core_dist_file)
     else:
-        pdm = dendropy.PhylogeneticDistanceMatrix.from_csv(src=open(core_dist_file),
-                                                           delimiter=",",
-                                                           is_first_row_column_names=True,
-                                                           is_first_column_row_names=False)
-        tree = pdm.nj_tree()
-
-    # Midpoint root tree and write outout
-    tree.reroot_at_midpoint(update_bipartitions=True, suppress_unifurcations=False)
-
-    # remove file as it can be large
-    os.remove(core_dist_file)
-
-    # return Newick string
-    tree_string = tree.as_string(schema="newick",
-                                 suppress_rooting=True,
-                                 unquoted_underscores=True)
-    tree_string = tree_string.replace("'","")
-    
+        matrix = []
+        for row, idx in enumerate(coreMat):
+            matrix.append(row[0:idx].tolist())
+        pdm = DistanceMatrix(seqLabels, matrix)
+        tree = DistanceTreeConstructor().nj(pdm)
+
+    tree.root_at_midpoint()
+    tree_string = tree_as_string(tree)
+
     return tree_string
 
 def mst_to_phylogeny(mst_network, names, use_gpu = False):
@@ -197,26 +210,22 @@ def mst_to_phylogeny(mst_network, names, use_gpu = False):
     #
     # MST graph -> phylogeny
     #
-    
+
     use_gpu = check_and_set_gpu(use_gpu, gpu_lib)
-    
-    # Define sequences names for tree
-    taxon_namespace = dendropy.TaxonNamespace(names)
-    # Initialise tree and create nodes
-    tree = dendropy.Tree(taxon_namespace=taxon_namespace)
+
 
     # Identify edges
     if use_gpu:
-        tree_nodes = {v:dendropy.Node(taxon=taxon_namespace[int(v)]) \
-            for v in range(0,mst_network.number_of_vertices())}
+        tree_nodes = [Node(label=v) for v in names]
         mst_edges_df = mst_network.view_edge_list()
     else:
-        tree_nodes = {v:dendropy.Node(taxon=taxon_namespace[int(v)]) for v in mst_network.get_vertices()}
+        tree_nodes = [Node(label=names[int(v)]) for v in mst_network.get_vertices()]
         mst_edges_df = pd.DataFrame(mst_network.get_edges(),
                                     columns = ['src', 'dst'])
         mst_edges_df['weights'] = list(mst_network.ep['weight'])
     seed_node_index = int(mst_edges_df[['src','dst']].stack().mode()[0])
-    tree.seed_node = tree_nodes[seed_node_index]
+    tree = Tree()
+    tree.root = tree_nodes[seed_node_index]
 
     # Generate links of tree
     parent_node_indices = [seed_node_index]
@@ -227,27 +236,24 @@ def mst_to_phylogeny(mst_network, names, use_gpu = False):
         mst_links_dst = mst_edges_df[['dst','weights']].loc[mst_edges_df['src']==parent_node_indices[i]]
         mst_links_src = mst_edges_df[['src','weights']].loc[mst_edges_df['dst']==parent_node_indices[i]]
         mst_links_src.columns = ['dst','weights']
-        mst_links = mst_links_dst.append(mst_links_src)
+        mst_links = pd.concat([mst_links_dst, mst_links_src])
         if use_gpu:
             mst_links = mst_links.to_pandas()
-        for (child_node,edge_length) in mst_links.itertuples(index=False, name=None):
+        for (child_node, edge_length) in mst_links.itertuples(index=False, name=None):
             if child_node not in added_nodes:
                 tree_nodes[parent_node_indices[i]].add_child(tree_nodes[child_node])
-                tree_nodes[child_node].edge_length = edge_length
+                tree_nodes[child_node].set_edge_length(edge_length)
                 added_nodes.add(child_node)
                 parent_node_indices.append(child_node)
         i = i + 1
 
     # Add zero length branches for internal nodes in MST
-    for node in tree.preorder_node_iter():
+    for node in tree.traverse_preorder():
         if not node.is_leaf():
-            new_child = dendropy.Node(taxon=node.taxon, edge_length=0.0)
-            node.taxon = None
+            new_child = Node(label=node.label, edge_length=0.0)
+            node.set_label(None)
             node.add_child(new_child)
 
     # Return tree as string
-    tree_string = tree.as_string(schema="newick",
-                                 suppress_rooting=True,
-                                 unquoted_underscores=True)
-                                     
+    tree_string = tree.newick()
     return tree_string
diff --git a/PopPUNK/tsne.py b/PopPUNK/tsne.py
deleted file mode 100644
index 75234037..00000000
--- a/PopPUNK/tsne.py
+++ /dev/null
@@ -1,137 +0,0 @@
-#!/usr/bin/env python
-# vim: set fileencoding=<utf-8> :
-# Copyright 2018-2020 John Lees and Nick Croucher
-
-import os
-import sys
-import numpy as np
-from sklearn import manifold as manifold_cpu
-
-from .utils import readPickle
-from .utils import check_and_set_gpu
-
-# Load GPU libraries
-try:
-    import cupyx
-    import cugraph
-    import cudf
-    import cupy as cp
-    from cuml import manifold as manifold_gpu
-    from numba import cuda
-    import rmm
-    gpu_lib = True
-except ImportError as e:
-    gpu_lib = False
-
-def generate_tsne(seqLabels, accMat, perplexity, outPrefix, overwrite, verbosity = 0, use_gpu = False):
-    """Generate t-SNE projection using accessory distances
-
-    Writes a plot of t-SNE clustering of accessory distances (.dot)
-
-    Args:
-        seqLabels (list)
-            Processed names of sequences being analysed.
-        accMat (numpy.array)
-            n x n array of accessory distances for n samples.
-        perplexity (int)
-            Perplexity parameter passed to t-SNE
-        outPrefix (str)
-            Prefix for all generated output files, which will be placed in
-            `outPrefix` subdirectory
-        overwrite (bool)
-            Overwrite existing output if present
-            (default = False)
-        verbosity (int)
-            Verbosity of t-SNE process (0-3)
-            (default = 0)
-        use_gpu (bool)
-            Whether to use GPU libraries
-    """
-    # generate accessory genome distance representation
-    tsne_filename = outPrefix + "/" + os.path.basename(outPrefix) + "_perplexity" + str(perplexity) + "_accessory_tsne.dot"
-    if overwrite or not os.path.isfile(tsne_filename):
-        sys.stderr.write("Running t-SNE\n")
-        use_gpu = check_and_set_gpu(use_gpu, gpu_lib)
-        if use_gpu:
-            accArray_embedded = manifold_gpu.TSNE(n_components=2, perplexity=perplexity, verbose=verbosity).fit_transform(np.array(accMat))
-        else:
-            accArray_embedded = manifold_cpu.TSNE(n_components=2, perplexity=perplexity, verbose=verbosity).fit_transform(np.array(accMat))
-
-        # print dot file
-        with open(tsne_filename, 'w') as nFile:
-            nFile.write("graph G { ")
-            for s, seqLabel in enumerate(seqLabels):
-                nFile.write('"' + seqLabel + '"' +
-                            '[x='+str(5*float(accArray_embedded[s][0]))+',y='+str(5*float(accArray_embedded[s][1]))+']; ')
-            nFile.write("}\n")
-    else:
-        sys.stderr.write("t-SNE analysis already exists; add --overwrite to replace\n")
-
-
-# command line parsing
-def get_options():
-
-    import argparse
-
-    parser = argparse.ArgumentParser(description='Run t-SNE projection with a specified perplexity',
-                                     prog='poppunk_tsne')
-
-    # input options
-    parser.add_argument('--distances', required=True, help='Prefix of input pickle and numpy file of pre-calculated '
-                                                           'distances')
-    parser.add_argument('--output', required=True, help='Name of output file')
-    parser.add_argument('--perplexity', help='Perplexity used to generate t-SNE projection [default = 30]', type=int, default=30)
-    parser.add_argument('--verbosity', help='Verbosity level for t-SNE (0-3) [default = 0]', type=int, default=0)
-    parser.add_argument('--use-gpu', help='Whether to use GPU libraries for t-SNE calculation', default = False, action='store_true')
-
-    return parser.parse_args()
-
-
-# main code
-def main():
-
-    # Check input ok
-    args = get_options()
-    verbosity = 0
-    if args.verbosity > 3:
-        verbosity = 3
-    elif args.verbosity > 0:
-        verbosity = args.verbosity
-
-    if not os.path.isdir(args.output):
-        try:
-            os.makedirs(args.output)
-        except OSError:
-            sys.stderr.write("Cannot create output directory\n")
-            sys.exit(1)
-
-    # load saved distance matrix
-    refList, queryList, self, distMat = readPickle(args.distances,
-                                                   enforce_self=True)
-
-    # process list of file names
-    seqLabels = [r.split('/')[-1].split('.')[0] for r in refList]
-
-    # generate accMat
-    accMat = np.zeros((len(seqLabels), len(seqLabels)), dtype=distMat.dtype)
-    i = 0
-    j = 1
-    # ref v ref (used for --create-db)
-    for row in distMat:
-        accMat[i, j] = row[1]
-        accMat[j, i] = accMat[i, j]
-
-        if j == len(refList) - 1:
-            i += 1
-            j = i + 1
-        else:
-            j += 1
-
-    # generate accessory genome distance representation
-    generate_tsne(seqLabels, accMat, args.perplexity, args.output, overwrite = True, use_gpu = args.use_gpu, verbosity = verbosity)
-
-
-if __name__ == "__main__":
-    main()
-
-    sys.exit(0)
diff --git a/PopPUNK/utils.py b/PopPUNK/utils.py
index c8f39d7d..2ac6c755 100644
--- a/PopPUNK/utils.py
+++ b/PopPUNK/utils.py
@@ -8,7 +8,6 @@
 import sys
 # additional
 import pickle
-import subprocess
 from collections import defaultdict
 from itertools import chain
 from tempfile import mkstemp
@@ -17,7 +16,6 @@
 
 import numpy as np
 import pandas as pd
-import h5py
 
 try:
     import cudf
@@ -58,9 +56,35 @@ def set_env(**environ):
         os.environ.clear()
         os.environ.update(old_environ)
 
+# https://stackoverflow.com/a/17954769
+from contextlib import contextmanager
+@contextmanager
+def stderr_redirected(to=os.devnull):
+    '''
+    import os
+
+    with stdout_redirected(to=filename):
+        print("from Python")
+        os.system("echo non-Python applications are also supported")
+    '''
+    fd = sys.stderr.fileno()
+
+    def _redirect_stderr(to):
+        sys.stderr.close()
+        os.dup2(to.fileno(), fd)
+        sys.stderr = os.fdopen(fd, 'w')
+
+    with os.fdopen(os.dup(fd), 'w') as old_stderr:
+        with open(to, 'w') as file:
+            _redirect_stderr(to=file)
+        try:
+            yield
+        finally:
+            _redirect_stderr(to=old_stderr)
+
 # Use partials to set up slightly different function calls between
 # both possible backends
-def setupDBFuncs(args, qc_dict):
+def setupDBFuncs(args):
     """Wraps common database access functions from sketchlib and mash,
     to try and make their API more similar
 
@@ -89,7 +113,6 @@ def setupDBFuncs(args, qc_dict):
                                 strand_preserved = args.strand_preserved,
                                 min_count = args.min_kmer_count,
                                 use_exact = args.exact_count,
-                                qc_dict = qc_dict,
                                 use_gpu = args.gpu_sketch,
                                 deviceid = args.deviceid)
     queryDatabase = partial(queryDatabaseSketchlib,
@@ -160,9 +183,9 @@ def readPickle(pklName, enforce_self=False, distances=True):
     """
     with open(pklName + ".pkl", 'rb') as pickle_file:
         rlist, qlist, self = pickle.load(pickle_file)
-        if enforce_self and not self:
+        if enforce_self and (not self or rlist != qlist):
             sys.stderr.write("Old distances " + pklName + ".npy not complete\n")
-            sys.stderr.exit(1)
+            sys.exit(1)
     if distances:
         X = np.load(pklName + ".npy")
     else:
@@ -235,114 +258,18 @@ def listDistInts(refSeqs, querySeqs, self=True):
         return comparisons
 
 
-def qcDistMat(distMat, refList, queryList, ref_db, prefix, qc_dict):
-    """Checks distance matrix for outliers.
-
-    Args:
-        distMat (np.array)
-            Core and accessory distances
-        refList (list)
-            Reference labels
-        queryList (list)
-            Query labels (or refList if self)
-        ref_db (str)
-            Prefix of reference database
-        prefix (str)
-            Prefix of output files
-        qc_dict (dict)
-            Dict of QC options
-
-    Returns:
-        seq_names_passing (list)
-            List of isolates passing QC distance filters
-        distMat ([n,2] numpy ndarray)
-            Filtered long form distance matrix
-    """
-
-    # avoid circular import
-    from .prune_db import prune_distance_matrix
-    from .sketchlib import removeFromDB
-    from .sketchlib import pickTypeIsolate
-
-    # Create overall list of sequences
-    if refList == queryList:
-        seq_names_passing = refList
-    else:
-        seq_names_passing = refList + queryList
-
-    # Sequences to remove
-    to_prune = []
-
-    # Create output directory if it does not exist already
-    if not os.path.isdir(prefix):
-        try:
-            os.makedirs(prefix)
-        except OSError:
-            sys.stderr.write("Cannot create output directory " + prefix + "\n")
-            sys.exit(1)
-
-    # Pick type isolate if not supplied
-    if qc_dict['type_isolate'] is None:
-        qc_dict['type_isolate'] = pickTypeIsolate(ref_db, seq_names_passing)
-        sys.stderr.write('Selected type isolate for distance QC is ' + qc_dict['type_isolate'] + '\n')
-
-    # First check with numpy, which is quicker than iterating over everything
-    #long_distance_rows = np.where([(distMat[:, 0] > qc_dict['max_pi_dist']) | (distMat[:, 1] > qc_dict['max_a_dist'])])[1].tolist()
-    long_distance_rows = np.where([(distMat[:, 0] > qc_dict['max_pi_dist']) | (distMat[:, 1] > qc_dict['max_a_dist'])],0,1)[0].tolist()
-    long_edges = poppunk_refine.generateTuples(long_distance_rows,
-                                                0,
-                                                self = (refList == queryList),
-                                                num_ref = len(refList),
-                                                int_offset = 0)
-    if len(long_edges) > 0:
-        # Prune sequences based on reference sequence
-        for (s,t) in long_edges:
-            if seq_names_passing[s] == qc_dict['type_isolate']:
-                to_prune.append(seq_names_passing[t])
-            elif seq_names_passing[t] == qc_dict['type_isolate']:
-                to_prune.append(seq_names_passing[s])
-
-    # prune based on distance from reference if provided
-    if qc_dict['qc_filter'] == 'stop' and len(to_prune) > 0:
-        sys.stderr.write('Outlier distances exceed QC thresholds; prune sequences or raise thresholds\n')
-        sys.stderr.write('Problem distances involved sequences ' + ';'.join(to_prune) + '\n')
-        sys.exit(1)
-    elif qc_dict['qc_filter'] == 'prune' and len(to_prune) > 0:
-        if qc_dict['type_isolate'] is None:
-            sys.stderr.write('Distances exceeded QC thresholds but no reference isolate supplied\n')
-            sys.stderr.write('Problem distances involved sequences ' + ';'.join(to_prune) + '\n')
-            sys.exit(1)
-        else:
-            # Remove sketches
-            db_name = ref_db + '/' + os.path.basename(ref_db) + '.h5'
-            filtered_db_name = prefix + '/' + 'filtered.' + os.path.basename(prefix) + '.h5'
-            removeFromDB(db_name,
-                         filtered_db_name,
-                         to_prune,
-                         full_names = True)
-            os.rename(filtered_db_name, db_name)
-            # Remove from distance matrix
-            seq_names_passing, distMat = prune_distance_matrix(seq_names_passing,
-                                                                to_prune,
-                                                                distMat,
-                                                                prefix + "/" + os.path.basename(prefix) + ".dists")
-            # Remove from reflist
-            sys.stderr.write('Pruned from the database after failing distance QC: ' + ';'.join(to_prune) + '\n')
-    else:
-        storePickle(seq_names_passing, seq_names_passing, True, distMat, prefix + "/" + os.path.basename(prefix) + ".dists")
-
-    return seq_names_passing, distMat
-
-
 def readIsolateTypeFromCsv(clustCSV, mode = 'clusters', return_dict = False):
     """Read cluster definitions from CSV file.
 
     Args:
         clustCSV (str)
             File name of CSV with isolate assignments
+        mode (str)
+            Type of file to read 'clusters', 'lineages', or 'external'
         return_type (str)
             If True, return a dict with sample->cluster instead
             of sets
+            [default = False]
 
     Returns:
         clusters (dict)
@@ -451,17 +378,19 @@ def update_distance_matrices(refList, distMat, queryList = None, query_ref_distM
         seqLabels = seqLabels + queryList
 
     if queryList == None:
-        coreMat = pp_sketchlib.longToSquare(distMat[:, [0]], threads)
-        accMat = pp_sketchlib.longToSquare(distMat[:, [1]], threads)
+        coreMat = pp_sketchlib.longToSquare(distVec=distMat[:, [0]],
+                                            num_threads=threads)
+        accMat = pp_sketchlib.longToSquare(distVec=distMat[:, [1]],
+                                           num_threads=threads)
     else:
-        coreMat = pp_sketchlib.longToSquareMulti(distMat[:, [0]],
-                                                 query_ref_distMat[:, [0]],
-                                                 query_query_distMat[:, [0]],
-                                                 threads)
-        accMat = pp_sketchlib.longToSquareMulti(distMat[:, [1]],
-                                                 query_ref_distMat[:, [1]],
-                                                 query_query_distMat[:, [1]],
-                                                 threads)
+        coreMat = pp_sketchlib.longToSquareMulti(distVec=distMat[:, [0]],
+                                                 query_ref_distVec=query_ref_distMat[:, [0]],
+                                                 query_query_distVec=query_query_distMat[:, [0]],
+                                                 num_threads=threads)
+        accMat = pp_sketchlib.longToSquareMulti(distVec=distMat[:, [1]],
+                                                query_ref_distVec=query_ref_distMat[:, [1]],
+                                                query_query_distVec=query_query_distMat[:, [1]],
+                                                num_threads=threads)
 
     # return outputs
     return seqLabels, coreMat, accMat
diff --git a/PopPUNK/visualise.py b/PopPUNK/visualise.py
index f60333a2..0237efc4 100644
--- a/PopPUNK/visualise.py
+++ b/PopPUNK/visualise.py
@@ -5,7 +5,6 @@
 # universal
 import os
 import sys
-import pickle
 # additional
 import numpy as np
 from scipy import sparse
@@ -117,15 +116,16 @@ def get_options():
     faGroup.add_argument('--cytoscape', help='Generate network output files for Cytoscape', default=False, action='store_true')
     faGroup.add_argument('--phandango', help='Generate phylogeny and TSV for Phandango visualisation', default=False, action='store_true')
     faGroup.add_argument('--grapetree', help='Generate phylogeny and CSV for grapetree visualisation', default=False, action='store_true')
-    faGroup.add_argument('--tree', help='Type of tree to calculate [default = nj]', type=str, default='nj',
+    faGroup.add_argument('--tree', help='Type of tree to calculate (not for cytoscape) [default = nj]', type=str, default='nj',
         choices=['nj', 'mst', 'both', 'none'])
     faGroup.add_argument('--mst-distances', help='Distances used to calculate a minimum spanning tree [default = core]', type=str,
         default='core', choices=accepted_weights_types)
     faGroup.add_argument('--rapidnj', help='Path to rapidNJ binary to build NJ tree for Microreact', default='rapidnj')
 
+    faGroup.add_argument('--api-key', help='API key for www.microreact.org, to directly create a visualisation', default=None)
     faGroup.add_argument('--perplexity',
                          type=float, default = 20.0,
-                         help='Perplexity used to calculate t-SNE projection (with --microreact) [default=20.0]')
+                         help='Perplexity used to calculate mandrake projection (with --microreact) [default=20.0]')
     faGroup.add_argument('--info-csv',
                          help='Epidemiological information CSV formatted for microreact (can be used with other outputs)')
 
@@ -180,18 +180,15 @@ def generate_visualisations(query_db,
                             gpu_graph,
                             info_csv,
                             rapidnj,
+                            api_key,
                             tree,
                             mst_distances,
                             overwrite,
-                            core_only,
-                            accessory_only,
-                            display_cluster,
-                            web):
+                            display_cluster):
 
     from .models import loadClusterFit
 
     from .network import construct_network_from_assignments
-    from .network import fetchNetwork
     from .network import generate_minimum_spanning_tree
     from .network import load_network_file
     from .network import cugraph_to_graph_tool
@@ -203,14 +200,11 @@ def generate_visualisations(query_db,
     from .plot import outputsForCytoscape
     from .plot import outputsForPhandango
     from .plot import outputsForGrapetree
-    from .plot import writeClusterCsv
-
-    from .prune_db import prune_distance_matrix
+    from .plot import createMicroreact
 
     from .sketchlib import readDBParams
-    from .sketchlib import getKmersFromReferenceDatabase
     from .sketchlib import addRandom
-    
+
     from .sparse_mst import generate_mst_from_sparse_input
 
     from .trees import load_tree, generate_nj_tree, mst_to_phylogeny
@@ -221,7 +215,6 @@ def generate_visualisations(query_db,
     from .utils import update_distance_matrices
     from .utils import readIsolateTypeFromCsv
     from .utils import joinClusterDicts
-    from .utils import listDistInts
     from .utils import read_rlist_from_distance_pickle
 
     #******************************#
@@ -237,6 +230,12 @@ def generate_visualisations(query_db,
     if not (microreact or phandango or grapetree or cytoscape):
         sys.stderr.write("Must specify at least one type of visualisation to output\n")
         sys.exit(1)
+    if cytoscape and not (microreact or phandango or grapetree):
+        if rank_fit == None and not os.path.isfile(network_file):
+            sys.stderr.write("For cytoscape, specify either a network file to visualise "
+                             "with --network-file or a lineage model with --rank-fit\n")
+            sys.exit(1)
+        tree = 'none'
 
     # make directory for new output files
     if not os.path.isdir(output):
@@ -262,9 +261,10 @@ def generate_visualisations(query_db,
         distances = distances
 
     # Determine whether to use sparse distances
+    combined_seq = None
     use_sparse = False
     use_dense = False
-    if (tree == 'mst' or tree == 'both' or cytoscape) and rank_fit is not None:
+    if (tree == "mst" or tree == "both") and rank_fit is not None:
         # Set flag
         use_sparse = True
         # Read list of sequence names and sparse distance matrix
@@ -278,17 +278,19 @@ def generate_visualisations(query_db,
         elif previous_mst is not None:
             sys.stderr.write('The prefix of the distance files used to create the previous MST'
                              ' is needed to use the network')
-    if tree == 'nj' or tree == 'both' or microreact:
+    if (tree == "nj" or tree == "both") or rank_fit == None:
         use_dense = True
         # Process dense distance matrix
         rlist, qlist, self, complete_distMat = readPickle(distances)
         if not self:
             qr_distMat = complete_distMat
+            combined_seq = rlist + qlist
         else:
             rr_distMat = complete_distMat
+            combined_seq = rlist
 
         # Fill in qq-distances if required
-        if self == False:
+        if self == False and tree != "none":
             sys.stderr.write("Note: Distances in " + distances + " are from assign mode\n"
                              "Note: Distance will be extended to full all-vs-all distances\n"
                              "Note: Re-run poppunk_assign with --update-db to avoid this\n")
@@ -301,22 +303,30 @@ def generate_visualisations(query_db,
             addRandom(query_db, qlist, kmers,
                       strand_preserved = strand_preserved, threads = threads)
             query_db_loc = query_db + "/" + os.path.basename(query_db)
-            qq_distMat = pp_sketchlib.queryDatabase(query_db_loc, query_db_loc,
-                                                    qlist, qlist, kmers,
-                                                    True, False,
-                                                    threads,
-                                                    gpu_dist,
-                                                    deviceid)
+            qq_distMat = pp_sketchlib.queryDatabase(ref_db_name=query_db_loc,
+                                                    query_db_name=query_db_loc,
+                                                    rList=qlist,
+                                                    qList=qlist,
+                                                    klist=kmers,
+                                                    random_correct=True,
+                                                    jaccard=False,
+                                                    num_threads=threads,
+                                                    use_gpu=gpu_dist,
+                                                    device_id=deviceid)
 
             # If the assignment was run with references, qrDistMat will be incomplete
             if rlist != rlist_original:
                 rlist = rlist_original
-                qr_distMat = pp_sketchlib.queryDatabase(ref_db_loc, query_db_loc,
-                                                        rlist, qlist, kmers,
-                                                        True, False,
-                                                        threads,
-                                                        gpu_dist,
-                                                        deviceid)
+                qr_distMat = pp_sketchlib.queryDatabase(ref_db_name=ref_db_loc,
+                                                        query_db_name=query_db_loc,
+                                                        rList=rlist,
+                                                        qList=qlist,
+                                                        klist=kmers,
+                                                        random_correct=True,
+                                                        jaccard=False,
+                                                        num_threads=threads,
+                                                        use_gpu=gpu_dist,
+                                                        device_id=deviceid)
 
         else:
             qlist = None
@@ -336,6 +346,7 @@ def generate_visualisations(query_db,
     #*******************************#
 
     # extract subset of distances if requested
+    all_seq = combined_seq
     if include_files is not None:
         viz_subset = set()
         with open(include_files, 'r') as assemblyFiles:
@@ -440,89 +451,90 @@ def generate_visualisations(query_db,
     mst_tree = None
     mst_graph = None
     nj_tree = None
-    if len(combined_seq) >= 3:
-        # MST tree
-        if tree == 'mst' or tree == 'both':
-            existing_tree = None
-            if not overwrite:
-                existing_tree = load_tree(output, "MST", distances=mst_distances)
-            if existing_tree is None:
-                # Check selecting clustering type is in CSV
-                clustering_name = 'Cluster'
-                if display_cluster != None:
-                    if display_cluster not in isolateClustering.keys():
+    if tree != 'none':
+        if len(combined_seq) >= 3:
+            # MST tree
+            if tree == 'mst' or tree == 'both':
+                existing_tree = None
+                if not overwrite:
+                    existing_tree = load_tree(output, "MST", distances=mst_distances)
+                if existing_tree is None:
+                    # Check selecting clustering type is in CSV
+                    clustering_name = 'Cluster'
+                    if display_cluster != None:
+                        if display_cluster not in isolateClustering.keys():
+                            clustering_name = list(isolateClustering.keys())[0]
+                            sys.stderr.write('Unable to find clustering column ' + display_cluster + ' in file ' +
+                                            prev_clustering + '; instead using ' + clustering_name + '\n')
+                        else:
+                            clustering_name = display_cluster
+                    else:
                         clustering_name = list(isolateClustering.keys())[0]
-                        sys.stderr.write('Unable to find clustering column ' + display_cluster + ' in file ' +
-                                         prev_clustering + '; instead using ' + clustering_name + '\n')
+                    if use_sparse:
+                        G = generate_mst_from_sparse_input(sparse_mat,
+                                                            rlist,
+                                                            old_rlist = old_rlist,
+                                                            previous_mst = previous_mst,
+                                                            gpu_graph = gpu_graph)
+                    elif use_dense:
+                        # Get distance matrix
+                        complete_distMat = \
+                            np.hstack((pp_sketchlib.squareToLong(core_distMat, threads).reshape(-1, 1),
+                                    pp_sketchlib.squareToLong(acc_distMat, threads).reshape(-1, 1)))
+                        # Dense network may be slow
+                        sys.stderr.write("Generating MST from dense distances (may be slow)\n")
+                        G = construct_network_from_assignments(combined_seq,
+                                                                combined_seq,
+                                                                [0]*complete_distMat.shape[0],
+                                                                within_label = 0,
+                                                                distMat = complete_distMat,
+                                                                weights_type = mst_distances,
+                                                                use_gpu = gpu_graph,
+                                                                summarise = False)
+                        if gpu_graph:
+                            G = cugraph.minimum_spanning_tree(G, weight='weights')
                     else:
-                        clustering_name = display_cluster
-                else:
-                    clustering_name = list(isolateClustering.keys())[0]
-                if use_sparse:
-                    G = generate_mst_from_sparse_input(sparse_mat,
-                                                        rlist,
-                                                        old_rlist = old_rlist,
-                                                        previous_mst = previous_mst,
-                                                        gpu_graph = gpu_graph)
-                elif use_dense:
-                    # Get distance matrix
-                    complete_distMat = \
-                        np.hstack((pp_sketchlib.squareToLong(core_distMat, threads).reshape(-1, 1),
-                                pp_sketchlib.squareToLong(acc_distMat, threads).reshape(-1, 1)))
-                    # Dense network may be slow
-                    sys.stderr.write("Generating MST from dense distances (may be slow)\n")
-                    G = construct_network_from_assignments(combined_seq,
-                                                            combined_seq,
-                                                            [0]*complete_distMat.shape[0],
-                                                            within_label = 0,
-                                                            distMat = complete_distMat,
-                                                            weights_type = mst_distances,
-                                                            use_gpu = gpu_graph,
-                                                            summarise = False)
+                        sys.stderr.write("Need either sparse or dense distances matrix to construct MST\n")
+                        exit(1)
+                    mst_graph = generate_minimum_spanning_tree(G, gpu_graph)
+                    del G
+                    # save outputs
+                    save_network(mst_graph,
+                                    prefix = output,
+                                    suffix = '_mst',
+                                    use_graphml = False,
+                                    use_gpu = gpu_graph)
                     if gpu_graph:
-                        G = cugraph.minimum_spanning_tree(G, weight='weights')
+                        mst_graph = cugraph_to_graph_tool(mst_graph, isolateNameToLabel(combined_seq))
+                    else:
+                        vid = mst_graph.new_vertex_property('string',
+                                                    vals = isolateNameToLabel(combined_seq))
+                        mst_graph.vp.id = vid
+                    mst_as_tree = mst_to_phylogeny(mst_graph,
+                                                    isolateNameToLabel(combined_seq),
+                                                    use_gpu = False)
+                    mst_as_tree = mst_as_tree.replace("'","")
+                    with open(os.path.join(output,os.path.basename(output) + '_mst.nwk'),'w') as tree_out:
+                        tree_out.write(mst_as_tree)
+                    drawMST(mst_graph, output, isolateClustering, clustering_name, overwrite)
                 else:
-                    sys.stderr.write("Need either sparse or dense distances matrix to construct MST\n")
-                    exit(1)
-                mst_graph = generate_minimum_spanning_tree(G, gpu_graph)
-                del G
-                # save outputs
-                save_network(mst_graph,
-                                prefix = output,
-                                suffix = '_mst',
-                                use_graphml = False,
-                                use_gpu = gpu_graph)
-                if gpu_graph:
-                    mst_graph = cugraph_to_graph_tool(mst_graph, isolateNameToLabel(combined_seq))
+                    mst_tree = existing_tree
+
+            # Generate NJ tree
+            if tree == 'nj' or tree == 'both':
+                existing_tree = None
+                if not overwrite:
+                    existing_tree = load_tree(output, "NJ")
+                if existing_tree is None:
+                    nj_tree = generate_nj_tree(core_distMat,
+                                                combined_seq,
+                                                output,
+                                                rapidnj,
+                                                threads = threads)
                 else:
-                    vid = mst_graph.new_vertex_property('string',
-                                                vals = isolateNameToLabel(combined_seq))
-                    mst_graph.vp.id = vid
-                mst_as_tree = mst_to_phylogeny(mst_graph,
-                                                isolateNameToLabel(combined_seq),
-                                                use_gpu = False)
-                mst_as_tree = mst_as_tree.replace("'","")
-                with open(os.path.join(output,os.path.basename(output) + '_mst.nwk'),'w') as tree_out:
-                    tree_out.write(mst_as_tree)
-                drawMST(mst_graph, output, isolateClustering, clustering_name, overwrite)
-            else:
-                mst_tree = existing_tree
-
-        # Generate NJ tree
-        if tree == 'nj' or tree == 'both':
-            existing_tree = None
-            if not overwrite:
-                existing_tree = load_tree(output, "NJ")
-            if existing_tree is None:
-                nj_tree = generate_nj_tree(core_distMat,
-                                            combined_seq,
-                                            output,
-                                            rapidnj,
-                                            threads = threads)
-            else:
-                nj_tree = existing_tree
-    else:
-        sys.stderr.write("Fewer than three sequences, not drawing trees\n")
+                    nj_tree = existing_tree
+        else:
+            sys.stderr.write("Fewer than three sequences, not drawing trees\n")
 
     #****************#
     #*              *#
@@ -533,17 +545,24 @@ def generate_visualisations(query_db,
     # Now have all the objects needed to generate selected visualisations
     if microreact:
         sys.stderr.write("Writing microreact output\n")
-        outputsForMicroreact(combined_seq,
-                             isolateClustering,
-                             nj_tree,
-                             mst_tree,
-                             acc_distMat,
-                             perplexity,
-                             output,
-                             info_csv,
-                             queryList = qlist,
-                             overwrite = overwrite,
-                             use_gpu = gpu_graph)
+        microreact_files = outputsForMicroreact(combined_seq,
+                                                isolateClustering,
+                                                nj_tree,
+                                                mst_tree,
+                                                acc_distMat,
+                                                perplexity,
+                                                output,
+                                                info_csv,
+                                                queryList=qlist,
+                                                overwrite=overwrite,
+                                                n_threads=threads,
+                                                use_gpu=gpu_graph,
+                                                device_id=deviceid)
+        url = createMicroreact(output, microreact_files, api_key)
+        if url != None:
+            sys.stderr.write("Microreact: " + url + "\n")
+        else:
+            sys.stderr.write("Provide --api-key to create microreact automatically\n")
 
     if phandango:
         sys.stderr.write("Writing phandango output\n")
@@ -569,22 +588,29 @@ def generate_visualisations(query_db,
 
     if cytoscape:
         sys.stderr.write("Writing cytoscape output\n")
+        import graph_tool.all as gt
         if network_file is not None:
             genomeNetwork = load_network_file(network_file, use_gpu = gpu_graph)
+            if gpu_graph:
+                genomeNetwork = cugraph_to_graph_tool(genomeNetwork, isolateNameToLabel(all_seq))
+            # Hard delete from network to remove samples (mask doesn't work neatly)
+            if viz_subset is not None:
+                remove_list = []
+                for keep, idx in enumerate(row_slice):
+                    if not keep:
+                        remove_list.append(idx)
+                genomeNetwork.remove_vertex(remove_list)
         elif rank_fit is not None:
             genomeNetwork = sparse_mat_to_network(sparse_mat, combined_seq, use_gpu = gpu_graph)
         else:
-            sys.stderr.write('Cytoscape output requires a network file or lineage rank fit is provided\n')
+            sys.stderr.write('Cytoscape output requires a network file or lineage rank fit to be provided\n')
             sys.exit(1)
-        if gpu_graph:
-            genomeNetwork = cugraph_to_graph_tool(genomeNetwork, isolateNameToLabel(combined_seq))
         outputsForCytoscape(genomeNetwork,
                             mst_graph,
                             combined_seq,
                             isolateClustering,
                             output,
-                            info_csv,
-                            viz_subset = viz_subset)
+                            info_csv)
         if model.type == 'lineage':
             sys.stderr.write("Note: Only support for output of cytoscape graph at lowest rank\n")
 
@@ -620,13 +646,11 @@ def main():
                             args.gpu_graph,
                             args.info_csv,
                             args.rapidnj,
+                            args.api_key,
                             args.tree,
                             args.mst_distances,
                             args.overwrite,
-                            args.core_only,
-                            args.accessory_only,
-                            args.display_cluster,
-                            web = False)
+                            args.display_cluster)
 
 if __name__ == '__main__':
     main()
diff --git a/PopPUNK/web.py b/PopPUNK/web.py
index cd24eb07..97ca5442 100644
--- a/PopPUNK/web.py
+++ b/PopPUNK/web.py
@@ -1,7 +1,6 @@
-from types import SimpleNamespace
+from re import I
 import h5py
 import os
-import re
 import sys
 import numpy as np
 import pandas as pd
@@ -10,165 +9,7 @@
 import networkx as nx
 from networkx.readwrite import json_graph
 
-import subprocess
-import uuid
-import glob
-import atexit
-from flask import Flask, request, jsonify
-from flask_cors import CORS, cross_origin
 import json
-from flask_apscheduler import APScheduler
-
-from PopPUNK.assign import assign_query
-from PopPUNK.utils import setupDBFuncs
-from PopPUNK.visualise import generate_visualisations
-
-# data locations
-db_prefix = 'WebOutput'
-db_location = '/home/poppunk-usr/' + os.getenv('POPPUNK_DBS_LOC', 'poppunk_dbs')
-
-app = Flask(__name__, instance_relative_config=True)
-app.config.update(
-    TESTING=True,
-    SCHEDULER_API_ENABLED=True,
-    SECRET_KEY=os.environ.get('FLASK_SECRET_KEY')
-)
-CORS(app, expose_headers='Authorization')
-scheduler = APScheduler()
-
-@app.route('/')
-def api_up():
-    return 'PopPUNK.web running\n'
-
-@app.route('/upload', methods=['POST'])
-@cross_origin()
-def sketchAssign():
-    """Recieve sketch and respond with clustering information"""
-    if not request.json:
-        return "not a json post"
-    if request.json:
-        sketch_dict = request.json
-        # determine database to use
-        if sketch_dict["species"] == "S.pneumoniae":
-            species = 'Streptococcus pneumoniae'
-            species_db = db_location + "/GPS_v3_references"
-        args = default_options(species_db)
-
-        outdir = "/tmp/" + db_prefix + "_" + str(uuid.uuid4())
-        if not os.path.exists(outdir):
-            os.mkdir(outdir)
-
-        qc_dict = {'run_qc': False }
-        dbFuncs = setupDBFuncs(args.assign, qc_dict)
-
-        # assign query to strain
-        ClusterResult = assign_query(dbFuncs,
-                                    args.assign.ref_db,
-                                    args.assign.q_files,
-                                    outdir,
-                                    qc_dict,
-                                    args.assign.update_db,
-                                    args.assign.write_references,
-                                    args.assign.distances,
-                                    args.assign.threads,
-                                    args.assign.overwrite,
-                                    args.assign.plot_fit,
-                                    args.assign.graph_weights,
-                                    args.assign.max_a_dist,
-                                    args.assign.max_pi_dist,
-                                    args.assign.type_isolate,
-                                    args.assign.model_dir,
-                                    args.assign.strand_preserved,
-                                    args.assign.previous_clustering,
-                                    args.assign.external_clustering,
-                                    args.assign.core_only,
-                                    args.assign.accessory_only,
-                                    args.assign.gpu_sketch,
-                                    args.assign.gpu_dist,
-                                    args.assign.gpu_graph,
-                                    args.assign.deviceid,
-                                    args.assign.web,
-                                    sketch_dict["sketch"],
-                                    args.assign.save_partial_query_graph)
-        query, query_prevalence, clusters, prevalences, alias_dict, to_include = \
-            summarise_clusters(outdir, species, species_db)
-        colours = get_colours(query, clusters)
-        url = api(query, args.assign.ref_db)
-
-        # generate visualisations from assign output
-        if len(to_include) < 3:
-            args.visualise.microreact = False
-        generate_visualisations(outdir,
-                                species_db,
-                                None,
-                                args.visualise.threads,
-                                outdir,
-                                args.visualise.gpu_dist,
-                                args.visualise.deviceid,
-                                args.visualise.external_clustering,
-                                args.visualise.microreact,
-                                args.visualise.phandango,
-                                args.visualise.grapetree,
-                                args.visualise.cytoscape,
-                                args.visualise.perplexity,
-                                args.visualise.strand_preserved,
-                                outdir + "/include.txt",
-                                species_db,
-                                species_db + "/" + os.path.basename(species_db) + "_clusters.csv",
-                                args.visualise.previous_query_clustering,
-                                outdir + "/" + os.path.basename(outdir) + "_graph.gt",                                args.visualise.gpu_graph,
-                                args.visualise.info_csv,
-                                args.visualise.rapidnj,
-                                args.visualise.tree,
-                                args.visualise.mst_distances,
-                                args.visualise.overwrite,
-                                args.visualise.core_only,
-                                args.visualise.accessory_only,
-                                args.visualise.display_cluster,
-                                web=True)
-        networkJson = graphml_to_json(outdir)
-        if len(to_include) >= 3:
-            with open(os.path.join(outdir, os.path.basename(outdir) + "_core_NJ.nwk"), "r") as p:
-                phylogeny = p.read()
-        else:
-            phylogeny = "A tree cannot be built with fewer than 3 samples."
-
-        # Convert outputs to JSON for post to site
-        response = {"species":species,
-                    "prev":str(query_prevalence) + '%',
-                    "query":query,
-                    "clusters":clusters,
-                    "prevalences":prevalences,
-                    "colours":colours,
-                    "microreactUrl":url,
-                    "aliases":alias_dict,
-                    "network": networkJson,
-                    "phylogeny": phylogeny}
-        response = json.dumps(response)
-        subprocess.run(["rm", "-rf", outdir]) # shutil issues on azure
-        return jsonify(response)
-
-def default_options(species_db):
-    """Default options for WebAPI"""
-    with open(os.path.join(species_db, "args.txt")) as a:
-        args_json = a.read()
-    args = json.loads(args_json, object_hook=lambda d: SimpleNamespace(**d))
-    args.assign.ref_db = species_db
-    args.assign.previous_clustering = species_db
-    args.assign.model_dir = species_db
-    args.assign.distances = os.path.join(species_db, os.path.basename(species_db) + ".dists")
-    args.assign.q_files = os.path.join(args.assign.output, "queries.txt")
-    return (args)
-
-def get_colours(query, clusters):
-    """Colour array for Plotly.js"""
-    colours = []
-    for clus in clusters:
-        if str(clus) == str(query):
-            colours.append('rgb(255,128,128)')
-        else:
-            colours.append('blue')
-    return colours
 
 def sketch_to_hdf5(sketches_dict, output):
     """Convert dict of JSON sketches to query hdf5 database"""
@@ -180,7 +21,10 @@ def sketch_to_hdf5(sketches_dict, output):
         qNames.append(top_key)
         kmers = []
         dists = []
-        sketch_dict = json.loads(top_value)
+        if type(top_value) == str:
+            sketch_dict = json.loads(top_value)
+        else:
+            sketch_dict = top_value
         sketch_props = sketches.create_group(top_key)
 
         for key, value in sketch_dict.items():
@@ -255,6 +99,7 @@ def highlight_cluster(query, cluster):
         colour = "blue"
     return colour
 
+# TODO: update to new microreact API
 def api(query, ref_db):
     """Post cluster and tree information to microreact"""
     url = "https://microreact.org/api/project/"
@@ -322,19 +167,8 @@ def summarise_clusters(output, species, species_db, qNames):
     if os.path.isfile(os.path.join(species_db, "aliases.csv")):
         aliasDF = pd.read_csv(os.path.join(species_db, "aliases.csv"))
         alias_dict = get_aliases(aliasDF, list(clusterDF['Taxon']), species)
-    else: 
+    else:
         alias_dict = {"Aliases": "NA"}
     return queries_names, queries_clusters, queries_prevalence, clusters, prevalences, alias_dict, to_include
 
-@scheduler.task('interval', id='clean_tmp', hours=1, misfire_grace_time=900)
-def clean_tmp():
-    print("Cleaning up unused databases")
-    for name in glob.glob("/tmp/" + db_prefix + "_*"):
-        shutil.rmtree(name)
 
-def main():
-    # data cleanup
-    scheduler.init_app(app)
-    scheduler.start()
-    atexit.register(lambda: scheduler.shutdown())
-    app.run(debug=False,use_reloader=False)
diff --git a/README.md b/README.md
index fa240387..4cf032bf 100644
--- a/README.md
+++ b/README.md
@@ -2,7 +2,7 @@
 
 <!-- badges: start -->
 [![Dev build Status](https://dev.azure.com/jlees/PopPUNK/_apis/build/status/johnlees.PopPUNK?branchName=master)](https://dev.azure.com/jlees/PopPUNK/_build/latest?definitionId=1&branchName=master)
-![Run tests](https://github.com/johnlees/PopPUNK/workflows/Run%20tests/badge.svg)
+![Run tests](https://github.com/bacpop/PopPUNK/workflows/Run%20tests/badge.svg)
 [![Documentation Status](https://readthedocs.org/projects/poppunk/badge/?version=latest)](https://poppunk.readthedocs.io/)
 [![Anaconda package](https://anaconda.org/bioconda/poppunk/badges/version.svg)](https://anaconda.org/bioconda/poppunk)
 [![PyPI version](https://badge.fury.io/py/poppunk.svg)](https://badge.fury.io/py/poppunk)
@@ -26,6 +26,18 @@ suggested methods paragraph.
 
 ## News
 
+### 2022-08-04
+The change in scikit-learn's API in v1.0.0 and above mean that HDBSCAN models
+fitted with `sklearn <=v0.24` will give an error when loaded. If you run into this,
+the solution is one of:
+- Downgrade sklearn to v0.24.
+- Run model refinement to turn your model into a boundary model instead (this will
+change clusters).
+- Refit your model in an environment with `sklearn >=v1.0`.
+
+If this is a common problem let us know, as we could write a script to 'upgrade'
+HDBSCAN models.
+See issue [#213](https://github.com/bacpop/PopPUNK/issues/213) for more details.
 ### 2021-03-15
 We have fixed a number of bugs with may affect the use of `poppunk_assign` with
 `--update-db`. We have also fixed a number of bugs with GPU distances. These are
@@ -34,14 +46,14 @@ We have fixed a number of bugs with may affect the use of `poppunk_assign` with
 ### 2020-09-30
 We have discovered a bug affecting the interaction of pp-sketchlib and PopPUNK.
 If you have used `PopPUNK >=v2.0.0` with `pp-sketchlib <v1.5.1` label order may
-be incorrect (see issue [#95](https://github.com/johnlees/PopPUNK/issues/95)).
+be incorrect (see issue [#95](https://github.com/bacpop/PopPUNK/issues/95)).
 
 Please upgrade to `PopPUNK >=v2.2` and `pp-sketchlib >=v1.5.1`. If this is not
 possible, you can either:
 - Run `scripts/poppunk_pickle_fix.py` on your `.dists.pkl` file and re-run
   model fits.
-- Create the database with `poppunk_sketch` directly, rather than `
-  PopPUNK --create-db`
+- Create the database with `poppunk_sketch` directly, rather than
+  `PopPUNK --create-db`
 
 ## Installation
 
@@ -68,3 +80,11 @@ conda config --add channels conda-forge
 
 See the [quickstart](https://poppunk.readthedocs.io/en/latest/quickstart.html) guide
 for a brief tutorial.
+
+## Docker image
+
+A docker image is available
+
+```
+docker pull mrcide/poppunk:bacpop-20
+```
diff --git a/boost/Dockerfile b/boost/Dockerfile
new file mode 100644
index 00000000..42272b09
--- /dev/null
+++ b/boost/Dockerfile
@@ -0,0 +1,5 @@
+FROM ubuntu:20.04
+
+RUN apt-get update && \
+    DEBIAN_FRONTEND=noninteractive apt-get -y install --no-install-recommends \
+    libboost-all-dev
diff --git a/boost/extract b/boost/extract
new file mode 100755
index 00000000..912e6b3d
--- /dev/null
+++ b/boost/extract
@@ -0,0 +1,15 @@
+#!/usr/bin/env bash
+set -eux
+HERE=$(dirname $0)
+ROOT=$(realpath $HERE/..)
+IMAGE=mrcide/poppunk:boost
+DEST=$ROOT/vendor
+REQUIRE=boost/sort/parallel_stable_sort/parallel_stable_sort.hpp
+docker build -t $IMAGE "$ROOT/boost"
+rm -rf $DEST/boost
+mkdir -p $DEST
+docker run \
+       --user "$(id -u):$(id -g)" \
+       -v $DEST:/dest \
+       $IMAGE \
+       bcp --boost=/usr/include $REQUIRE /dest
diff --git a/buildkite/pipeline.yml b/buildkite/pipeline.yml
new file mode 100644
index 00000000..5c4894a3
--- /dev/null
+++ b/buildkite/pipeline.yml
@@ -0,0 +1,13 @@
+steps:
+  - label: ":whale::python: Build"
+    command: docker2/build
+
+  - wait
+
+  - label: ":hammer: Test image"
+    command: docker2/test
+
+  - wait
+
+  - label: ":shipit: Push images"
+    command: docker2/push
diff --git a/docker2/Dockerfile b/docker2/Dockerfile
new file mode 100644
index 00000000..9f895932
--- /dev/null
+++ b/docker2/Dockerfile
@@ -0,0 +1,43 @@
+FROM python:3.10
+
+RUN apt-get update && \
+    apt-get install -y --no-install-recommends \
+                          cmake \
+                          gfortran \
+                          libarmadillo-dev \
+                          libeigen3-dev \
+                          libopenblas-dev \
+                          software-properties-common
+
+RUN git clone https://github.com/somme89/rapidNJ.git && \
+    cd rapidNJ && \
+    make && \
+    mv ./bin/rapidnj /usr/bin &&\
+    cd .. && rm -r rapidNJ
+
+# Special snowflake treatment for graph_tool because the maintainer
+# refuses to make the package pip-installable, meaning we can't depend
+# on it in a sane way. Least pain seems to be to install via apt, even
+# though that might result in slightly incorrect versions (built
+# around system python).
+#
+# The alternative would be to build it from source
+# https://git.skewed.de/count0/graph-tool/-/wikis/installation-instructions#manual-compilation
+# but it takes over an hour to compile apparently, so that sounds not
+# ideal.
+RUN add-apt-repository https://downloads.skewed.de/apt && \
+    apt-key adv --keyserver keyserver.ubuntu.com \
+            --recv-key 612DEFB798507F25 && \
+    apt-get update && \
+    apt-get install -y --no-install-recommends python3-graph-tool && \
+    ln -s /usr/lib/python3/dist-packages/graph_tool \
+      /usr/local/lib/python3.10/site-packages
+
+COPY requirements.txt /tmp/requirements.txt
+RUN pip install pybind11 && \
+  pip install -r /tmp/requirements.txt && \
+  rm -f /tmp/requirements.txt
+
+COPY . /src
+WORKDIR /src
+RUN python setup.py install
diff --git a/docker2/build b/docker2/build
new file mode 100755
index 00000000..68784d80
--- /dev/null
+++ b/docker2/build
@@ -0,0 +1,14 @@
+#!/usr/bin/env bash
+set -ex
+
+HERE=$(dirname $0)
+. $HERE/common
+
+docker build --pull \
+       --tag $TAG_SHA \
+       -f docker2/Dockerfile \
+       $PACKAGE_ROOT
+
+# We always push the SHA tagged versions, for debugging if the tests
+# after this step fail
+docker push $TAG_SHA
diff --git a/docker2/common b/docker2/common
new file mode 100755
index 00000000..a0448139
--- /dev/null
+++ b/docker2/common
@@ -0,0 +1,25 @@
+#!/usr/bin/env bash
+set -e
+
+PACKAGE_ROOT=$(realpath $HERE/..)
+PACKAGE_NAME=poppunk
+PACKAGE_ORG=mrcide
+
+# Buildkite doesn't check out a full history from the remote (just the
+# single commit) so you end up with a detached head and git rev-parse
+# doesn't work
+if [ "$BUILDKITE" = "true" ]; then
+    GIT_SHA=${BUILDKITE_COMMIT:0:7}
+else
+    GIT_SHA=$(git -C "$PACKAGE_ROOT" rev-parse --short=7 HEAD)
+fi
+
+if [ "$BUILDKITE" = "true" ]; then
+    GIT_BRANCH=$BUILDKITE_BRANCH
+else
+    GIT_BRANCH=$(git -C "$PACKAGE_ROOT" symbolic-ref --short HEAD)
+fi
+
+TAG_SHA="${PACKAGE_ORG}/${PACKAGE_NAME}:${GIT_SHA}"
+TAG_BRANCH="${PACKAGE_ORG}/${PACKAGE_NAME}:${GIT_BRANCH}"
+TAG_LATEST="${PACKAGE_ORG}/${PACKAGE_NAME}:latest"
diff --git a/docker2/push b/docker2/push
new file mode 100755
index 00000000..103533a1
--- /dev/null
+++ b/docker2/push
@@ -0,0 +1,16 @@
+#!/usr/bin/env bash
+set -e
+
+HERE=$(dirname $0)
+. $HERE/common
+
+# In case we switch agents between steps
+[ ! -z $(docker images -q $TAG_SHA) ] || docker pull $TAG_SHA
+
+docker tag $TAG_SHA $TAG_BRANCH
+docker push $TAG_BRANCH
+
+if [ $GIT_BRANCH == "master" ]; then
+   docker tag $TAG_SHA $TAG_LATEST
+   docker push $TAG_LATEST
+fi
diff --git a/docs/api.rst b/docs/api.rst
index e2d26c9a..d5f7e3cc 100644
--- a/docs/api.rst
+++ b/docs/api.rst
@@ -31,6 +31,12 @@ Functions used to fit DBSCAN to a database. Access using
 .. automodule:: PopPUNK.dbscan
    :members:
 
+mandrake.py
+-----------
+
+.. automodule:: PopPUNK.mandrake
+   :members:
+
 models.py
 ---------
 
@@ -75,12 +81,6 @@ sketchlib.py
 .. automodule:: PopPUNK.sketchlib
    :members:
 
-tsne.py
--------
-
-.. automodule:: PopPUNK.tsne
-   :members:
-
 utils.py
 --------
 
diff --git a/docs/citing.rst b/docs/citing.rst
index b6193f5c..ab1b814a 100644
--- a/docs/citing.rst
+++ b/docs/citing.rst
@@ -36,5 +36,5 @@ gives:
     version 2.4.0 (doi:10.1101/gr.241455.118) [1-5].
 
 If your journal requires versions for all software packages, you may find running
-``conda list`` helpful. The ``poppunk_db_info.py`` script (:ref:`db-info`) can be
-run on your ``.h5`` files to give useful information too.
\ No newline at end of file
+``conda list`` helpful. Running ``poppunk_info`` (:ref:`db-info`) on your ``.h5``
+files will give useful information too.
diff --git a/docs/conf.py b/docs/conf.py
index b6376398..50d053ce 100644
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -31,7 +31,8 @@
 # Add any Sphinx extension module names here, as strings. They can be
 # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
 # ones.
-extensions = ['sphinx.ext.autodoc',
+extensions = [
+    'sphinx.ext.autodoc',
     'sphinx.ext.mathjax',
     'sphinx.ext.viewcode']
 
@@ -66,7 +67,7 @@
 
 # General information about the project.
 project = 'PopPUNK'
-copyright = '2018-2020, John Lees and Nicholas Croucher'
+copyright = '2018-2022, John Lees and Nicholas Croucher'
 author = 'John Lees, Daniel Anderson and Nicholas Croucher'
 
 # The version info for the project you're documenting, acts as replacement for
@@ -74,9 +75,9 @@
 # built documents.
 #
 # The short X.Y version.
-version = '2.3.0'
+version = '2.5.0'
 # The full version, including alpha/beta/rc tags.
-release = '2.3.0'
+release = '2.5.0'
 
 # The language for content autogenerated by Sphinx. Refer to documentation
 # for a list of supported languages.
diff --git a/docs/images/contam_DPGMM_better_fit.png b/docs/images/contam_DPGMM_better_fit.png
deleted file mode 100644
index a82af24b..00000000
Binary files a/docs/images/contam_DPGMM_better_fit.png and /dev/null differ
diff --git a/docs/images/contam_DPGMM_fit.png b/docs/images/contam_DPGMM_fit.png
deleted file mode 100644
index 72fd0da7..00000000
Binary files a/docs/images/contam_DPGMM_fit.png and /dev/null differ
diff --git a/docs/images/cytoscape_contaminant.png b/docs/images/cytoscape_contaminant.png
deleted file mode 100644
index c6af3c37..00000000
Binary files a/docs/images/cytoscape_contaminant.png and /dev/null differ
diff --git a/docs/images/listeria_microreact.png b/docs/images/listeria_microreact.png
deleted file mode 100644
index 05296a1a..00000000
Binary files a/docs/images/listeria_microreact.png and /dev/null differ
diff --git a/docs/images/microreact.png b/docs/images/microreact.png
index f5c91507..f37a37b1 100644
Binary files a/docs/images/microreact.png and b/docs/images/microreact.png differ
diff --git a/docs/images/microreact_perplexity5.png b/docs/images/microreact_perplexity5.png
index 9f07be50..2c18b444 100644
Binary files a/docs/images/microreact_perplexity5.png and b/docs/images/microreact_perplexity5.png differ
diff --git a/docs/images/poppunk_flowchart.png b/docs/images/poppunk_flowchart.png
index 4a71d8d8..7aa72e1b 100644
Binary files a/docs/images/poppunk_flowchart.png and b/docs/images/poppunk_flowchart.png differ
diff --git a/docs/images/poppunk_logo_original.png b/docs/images/poppunk_logo_original.png
new file mode 100644
index 00000000..f29e7590
Binary files /dev/null and b/docs/images/poppunk_logo_original.png differ
diff --git a/docs/images/poppunk_v2.png b/docs/images/poppunk_v2.png
index 2d82e4fe..57b46982 100644
Binary files a/docs/images/poppunk_v2.png and b/docs/images/poppunk_v2.png differ
diff --git a/docs/images/poppunk_v2_tie.png b/docs/images/poppunk_v2_tie.png
new file mode 100644
index 00000000..d4bf0ea2
Binary files /dev/null and b/docs/images/poppunk_v2_tie.png differ
diff --git a/docs/index.rst b/docs/index.rst
index 9e3bcbf2..8493bcb9 100644
--- a/docs/index.rst
+++ b/docs/index.rst
@@ -1,13 +1,16 @@
 PopPUNK documentation
 ==================================
-.. image:: images/poppunk_v2.png
-   :alt:  PopPUNK (Population Partitioning Using Nucleotide K-mers)
-   :align: center
-
 PopPUNK is a tool for clustering genomes. We refer to the clusters as
 variable-length-k-mer clusters, or **VLKCs**. Biologically, these clusters
 typically represent distinct strains. We refer to subclusters of strains as lineages.
 
+If you are new to PopPUNK, we'd recommend starting on :doc:`installation`, then
+by reading the :doc:`overview`.
+
+.. image:: images/poppunk_v2.png
+   :alt:  PopPUNK (Population Partitioning Using Nucleotide K-mers)
+   :align: center
+
 The first version was targeted specifically
 as bacterial genomes, but the current version has also been used for viruses
 (e.g. enterovirus, influenza, SARS-CoV-2) and eukaryotes (e.g. *Candida* sp.,
@@ -17,8 +20,7 @@ core and accessory distances, and machine learning tools written in python to
 use these to cluster genomes. A detailed description of the method can be found
 in the `paper <https://doi.org/10.1101/gr.241455.118>`_.
 
-If you are new to PopPUNK, we'd recommend starting on :doc:`installation`, then
-by reading the :doc:`best_practises`.
+
 
 .. important::
    Looking for older versions of the documentation? For previous versions with
@@ -31,19 +33,18 @@ by reading the :doc:`best_practises`.
 
    self
    installation.rst
-   best_practises.rst
-   online.rst
-   query_assignment.rst
+   overview.rst
    sketching.rst
    qc.rst
    model_fitting.rst
    model_distribution.rst
+   query_assignment.rst
    visualisation.rst
    mst.rst
    subclustering.rst
+   online.rst
    gpu.rst
    troubleshooting.rst
-   options.rst
    scripts.rst
    citing.rst
    api.rst
diff --git a/docs/installation.rst b/docs/installation.rst
index 99c63533..cbc1915b 100644
--- a/docs/installation.rst
+++ b/docs/installation.rst
@@ -31,6 +31,8 @@ Then run::
 
     conda install poppunk
 
+If you want to use GPUs, take a look at :doc:`gpu`.
+
 If you are having conflict issues with conda, our advice would be:
 
 - Remove and reinstall miniconda.
@@ -49,7 +51,8 @@ If you do not have conda, you can also install through pip::
 
     python3 -m pip install poppunk
 
-This may not deal with all necessary :ref:`dependencies`.
+This may not deal with all necessary dependencies, but we are working on it
+and it should again be possible in an upcoming release.
 
 Clone the code
 --------------
@@ -60,27 +63,7 @@ You can also clone the github to run the latest version, which is executed by::
 
 This will also give access to the :ref:`scripts`.
 
-You will need to install the :ref:`dependencies` yourself (you can still use
-conda or pip for this purpose).
-
-.. _dependencies:
-
-Dependencies
-------------
-This documentation refers to a conda installation with the following packages:
-
-* python3 (3.8.2)
-* ``pp-sketchlib`` (1.6.2)
-* ``DendroPy`` (4.3.0)
-* ``hdbscan`` (0.8.13)
-* ``matplotlib`` (2.1.2)
-* ``graph-tool`` (2.31)
-* ``numpy`` (1.14.1)
-* ``pandas`` (0.22.0)
-* ``scikit-learn`` (0.19.1)
-* ``scipy`` (1.0.0)
-* ``sharedmem`` (0.3.5)
-
-Optionally, you can use `rapidnj <http://birc.au.dk/software/rapidnj/>`__
-if producing output with ``--microreact`` and ``--rapidnj`` options. We used
-v2.3.2.
+You will need to install the dependencies yourself (you can still use
+conda or pip for this purpose). See ``environment.yml``.
+
+
diff --git a/docs/miscellaneous.rst b/docs/miscellaneous.rst
index 133fd20b..48d72a1e 100644
--- a/docs/miscellaneous.rst
+++ b/docs/miscellaneous.rst
@@ -3,6 +3,14 @@ Miscellaneous
 
 Rejected/alternative logos
 ^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. image:: images/poppunk_logo_original.png
+   :alt:  Original logo
+   :align: center
+
 .. image:: images/poppunk.png
    :alt:  God save the queen
    :align: center
+
+.. image:: images/poppunk_v2_tie.png
+   :alt:  Avril
+   :align: center
\ No newline at end of file
diff --git a/docs/model_fitting.rst b/docs/model_fitting.rst
index fc5f6eb6..c38b8ca2 100644
--- a/docs/model_fitting.rst
+++ b/docs/model_fitting.rst
@@ -1,5 +1,5 @@
-Fitting new models
-==================
+Fitting new models (``--fit-model``)
+====================================
 
 .. |nbsp| unicode:: 0xA0
    :trim:
@@ -292,6 +292,12 @@ for small datasets with sparse points, or for datasets without much structure in
 and accessory), and that some points are classified as 'noise' so not all of their
 edges are included in the network (these are the small black points).
 
+.. warning::
+    HDBSCAN models are not backwards compatible from sklearn v1.0 onwards. We
+    would recommend using at least this version. Even better would be to then run
+    model refinement (:ref:`refine-models`) to get a simpler and faster model
+    for onward query assignment.
+
 dbscan usually needs little modification to run::
 
     poppunk --fit-model dbscan --ref-db listeria
@@ -400,6 +406,8 @@ Setting either ``--min-cluster-prop`` or ``--D`` too low can cause the fit to fa
 
     Failed to find distinct clusters in this dataset
 
+.. _refine-models:
+
 refine
 ------
 Model refinement is slightly different: it takes a model already fitted by :ref:`bgmm`
@@ -603,6 +611,7 @@ you can manually provide a file with ``--manual-start`` to give the starting par
 
     start 0,0
     end 0.5,0.6
+    scaled True
 
 A key, followed by its value (space separated).
 
@@ -610,6 +619,10 @@ A key, followed by its value (space separated).
 These define the two red points (and therefore the
 search range) in the output plot.
 
+``scaled`` defines whether these are on the [0,1] scale. If these have been set
+using means from the terminal output this should be ``True``. Otherwise, if you
+have set them based on the plot (unscaled space), set to ``False``.
+
 .. _indiv-refine:
 
 Using core/accessory only
diff --git a/docs/online.rst b/docs/online.rst
index 9c97dddb..68211610 100644
--- a/docs/online.rst
+++ b/docs/online.rst
@@ -4,6 +4,10 @@ PopPUNK-web is available at https://web.poppunk.net/.
 This interface has been designed for non-specialists and allows you to easily analyse and assign lineages to your query genomic sequences,
 using pre-optimised species-specific databases and default parameters.
 
+.. warning::
+   PopPUNK-web is temporarily down while we work on an updated and better supported
+   version (beebop).
+
 .. contents::
    :local:
 
diff --git a/docs/options.rst b/docs/options.rst
deleted file mode 100644
index fea404e5..00000000
--- a/docs/options.rst
+++ /dev/null
@@ -1,326 +0,0 @@
-Options
-=======
-
-**Contents**:
-
-.. contents::
-   :local:
-
-poppunk
--------
-
-Usage::
-
-       poppunk [-h]
-               (--create-db | --fit-model {bgmm,dbscan,refine,lineage,threshold} | --use-model)
-               [--ref-db REF_DB] [--r-files R_FILES]
-               [--distances DISTANCES]
-               [--external-clustering EXTERNAL_CLUSTERING]
-               [--output OUTPUT] [--plot-fit PLOT_FIT] [--overwrite]
-               [--graph-weights] [--min-k MIN_K] [--max-k MAX_K]
-               [--k-step K_STEP] [--sketch-size SKETCH_SIZE]
-               [--codon-phased] [--min-kmer-count MIN_KMER_COUNT]
-               [--exact-count] [--strand-preserved]
-               [--qc-filter {stop,prune,continue}] [--retain-failures]
-               [--max-a-dist MAX_A_DIST] [--length-sigma LENGTH_SIGMA]
-               [--length-range LENGTH_RANGE LENGTH_RANGE]
-               [--prop-n PROP_N] [--upper-n UPPER_N] [--K K] [--D D]
-               [--min-cluster-prop MIN_CLUSTER_PROP]
-               [--threshold THRESHOLD] [--pos-shift POS_SHIFT]
-               [--neg-shift NEG_SHIFT] [--manual-start MANUAL_START]
-               [--indiv-refine] [--no-local] [--model-dir MODEL_DIR]
-               [--ranks RANKS] [--use-accessory] [--threads THREADS]
-               [--gpu-sketch] [--gpu-dist] [--deviceid DEVICEID]
-               [--version]
-
-Command line options::
-
-  optional arguments:
-    -h, --help            show this help message and exit
-
-  Mode of operation:
-    --create-db           Create pairwise distances database between
-                          reference sequences
-    --fit-model {bgmm,dbscan,refine,lineage,threshold}
-                          Fit a mixture model to a reference database
-    --use-model           Apply a fitted model to a reference database to
-                          restore database files
-
-  Input files:
-    --ref-db REF_DB       Location of built reference database
-    --r-files R_FILES     File listing reference input assemblies
-    --distances DISTANCES
-                          Prefix of input pickle of pre-calculated distances
-    --external-clustering EXTERNAL_CLUSTERING
-                          File with cluster definitions or other labels
-                          generated with any other method.
-
-  Output options:
-    --output OUTPUT       Prefix for output files
-    --plot-fit PLOT_FIT   Create this many plots of some fits relating k-mer
-                          to core/accessory distances [default = 0]
-    --overwrite           Overwrite any existing database files
-    --graph-weights       Save within-strain Euclidean distances into the
-                          graph
-
-  Create DB options:
-    --min-k MIN_K         Minimum kmer length [default = 13]
-    --max-k MAX_K         Maximum kmer length [default = 29]
-    --k-step K_STEP       K-mer step size [default = 4]
-    --sketch-size SKETCH_SIZE
-                          Kmer sketch size [default = 10000]
-    --codon-phased        Used codon phased seeds X--X--X [default = False]
-    --min-kmer-count MIN_KMER_COUNT
-                          Minimum k-mer count when using reads as input
-                          [default = 0]
-    --exact-count         Use the exact k-mer counter with reads [default =
-                          use countmin counter]
-    --strand-preserved    Treat input as being on the same strand, and
-                          ignore reverse complement k-mers [default = use
-                          canonical k-mers]
-
-  Quality control options:
-    --qc-filter {stop,prune,continue}
-                          Behaviour following sequence QC step: "stop"
-                          [default], "prune" (analyse data passing QC), or
-                          "continue" (analyse all data)
-    --retain-failures     Retain sketches of genomes that do not pass QC
-                          filters in separate database [default = False]
-    --max-a-dist MAX_A_DIST
-                          Maximum accessory distance to permit [default =
-                          0.5]
-    --length-sigma LENGTH_SIGMA
-                          Number of standard deviations of length
-                          distribution beyond which sequences will be
-                          excluded [default = 5]
-    --length-range LENGTH_RANGE LENGTH_RANGE
-                          Allowed length range, outside of which sequences
-                          will be excluded [two values needed - lower and
-                          upper bounds]
-    --prop-n PROP_N       Threshold ambiguous base proportion above which
-                          sequences will be excluded [default = 0.1]
-    --upper-n UPPER_N     Threshold ambiguous base count above which
-                          sequences will be excluded
-
-  Model fit options:
-    --K K                 Maximum number of mixture components [default = 2]
-    --D D                 Maximum number of clusters in DBSCAN fitting
-                          [default = 100]
-    --min-cluster-prop MIN_CLUSTER_PROP
-                          Minimum proportion of points in a cluster in
-                          DBSCAN fitting [default = 0.0001]
-    --threshold THRESHOLD
-                          Cutoff if using --fit-model threshold
-
-  Refine model options:
-    --pos-shift POS_SHIFT
-                          Maximum amount to move the boundary away from
-                          origin [default = to between-strain mean]
-    --neg-shift NEG_SHIFT
-                          Maximum amount to move the boundary towards the
-                          origin [default = to within-strain mean]
-    --manual-start MANUAL_START
-                          A file containing information for a start point.
-                          See documentation for help.
-    --indiv-refine {both,core,accessory}
-                          Also run refinement for core and accessory
-                          individually
-    --no-local            Do not perform the local optimization step (speed
-                          up on very large datasets)
-    --model-dir MODEL_DIR
-                          Directory containing model to use for assigning
-                          queries to clusters [default = reference database
-                          directory]
-
-  Lineage analysis options:
-    --ranks RANKS         Comma separated list of ranks used in lineage
-                          clustering [default = 1,2,3]
-    --use-accessory       Use accessory distances for lineage definitions
-                          [default = use core distances]
-
-  Other options:
-    --threads THREADS     Number of threads to use [default = 1]
-    --gpu-sketch          Use a GPU when calculating sketches (read data
-                          only) [default = False]
-    --gpu-dist            Use a GPU when calculating distances [default =
-                          False]
-    --deviceid DEVICEID   CUDA device ID, if using GPU [default = 0]
-    --version             show program's version number and exit
-
-poppunk_assign
---------------
-
-Usage::
-
-  poppunk_assign [-h] --db DB --query QUERY [--distances DISTANCES]
-                        [--external-clustering EXTERNAL_CLUSTERING] --output
-                        OUTPUT [--plot-fit PLOT_FIT] [--write-references]
-                        [--update-db] [--overwrite] [--graph-weights]
-                        [--min-kmer-count MIN_KMER_COUNT] [--exact-count]
-                        [--strand-preserved] [--max-a-dist MAX_A_DIST]
-                        [--model-dir MODEL_DIR]
-                        [--previous-clustering PREVIOUS_CLUSTERING]
-                        [--core-only] [--accessory-only] [--threads THREADS]
-                        [--gpu-sketch] [--gpu-dist] [--deviceid DEVICEID]
-                        [--version]
-
-Command line options::
-
-  optional arguments:
-    -h, --help            show this help message and exit
-
-  Input files:
-    --db DB               Location of built reference database
-    --query QUERY         File listing query input assemblies
-    --distances DISTANCES
-                          Prefix of input pickle of pre-calculated distances
-                          (if not in --db)
-    --external-clustering EXTERNAL_CLUSTERING
-                          File with cluster definitions or other labels
-                          generated with any other method.
-
-  Output options:
-    --output OUTPUT       Prefix for output files (required)
-    --plot-fit PLOT_FIT   Create this many plots of some fits relating k-mer
-                          to core/accessory distances [default = 0]
-    --write-references    Write reference database isolates' cluster
-                          assignments out too
-    --update-db           Update reference database with query sequences
-    --overwrite           Overwrite any existing database files
-    --graph-weights       Save within-strain Euclidean distances into the
-                          graph
-
-  Kmer comparison options:
-    --min-kmer-count MIN_KMER_COUNT
-                          Minimum k-mer count when using reads as input
-                          [default = 0]
-    --exact-count         Use the exact k-mer counter with reads [default =
-                          use countmin counter]
-    --strand-preserved    Treat input as being on the same strand, and
-                          ignore reverse complement k-mers [default = use
-                          canonical k-mers]
-
-  Quality control options:
-    --max-a-dist MAX_A_DIST
-                          Maximum accessory distance to permit [default =
-                          0.5]
-
-  Database querying options:
-    --model-dir MODEL_DIR
-                          Directory containing model to use for assigning
-                          queries to clusters [default = reference database
-                          directory]
-    --previous-clustering PREVIOUS_CLUSTERING
-                          Directory containing previous cluster definitions
-                          and network [default = use that in the directory
-                          containing the model]
-    --core-only           (with a 'refine' model) Use a core-distance only
-                          model for assigning queries [default = False]
-    --accessory-only      (with a 'refine' or 'lineage' model) Use an
-                          accessory-distance only model for assigning
-                          queries [default = False]
-
-  Other options:
-    --threads THREADS     Number of threads to use [default = 1]
-    --gpu-sketch          Use a GPU when calculating sketches (read data
-                          only) [default = False]
-    --gpu-dist            Use a GPU when calculating distances [default =
-                          False]
-    --deviceid DEVICEID   CUDA device ID, if using GPU [default = 0]
-    --version             show program's version number and exit
-
-poppunk_visualise
------------------
-
-Usage::
-
-  poppunk_visualise [-h] --ref-db REF_DB [--query-db QUERY_DB]
-                          [--distances DISTANCES]
-                          [--include-files INCLUDE_FILES]
-                          [--external-clustering EXTERNAL_CLUSTERING]
-                          [--model-dir MODEL_DIR]
-                          [--previous-clustering PREVIOUS_CLUSTERING]
-                          [--previous-query-clustering PREVIOUS_QUERY_CLUSTERING]
-                          --output OUTPUT [--overwrite] [--core-only]
-                          [--accessory-only] [--microreact]
-                          [--cytoscape] [--phandango] [--grapetree]
-                          [--tree {nj,mst,both}]
-                          [--mst-distances {core,accessory,euclidean}]
-                          [--rapidnj RAPIDNJ] [--perplexity PERPLEXITY]
-                          [--info-csv INFO_CSV] [--threads THREADS]
-                          [--gpu-dist] [--deviceid DEVICEID]
-                          [--strand-preserved] [--version]
-
-Command line options::
-
-  optional arguments:
-    -h, --help            show this help message and exit
-
-  Input files:
-    --ref-db REF_DB       Location of built reference database
-    --query-db QUERY_DB   Location of query database, if distances are
-                          from ref-query
-    --distances DISTANCES
-                          Prefix of input pickle of pre-calculated
-                          distances
-    --include-files INCLUDE_FILES
-                          File with list of sequences to include in
-                          visualisation. Default is to use all sequences
-                          in database.
-    --external-clustering EXTERNAL_CLUSTERING
-                          File with cluster definitions or other labels
-                          generated with any other method.
-    --model-dir MODEL_DIR
-                          Directory containing model to use for
-                          assigning queries to clusters [default =
-                          reference database directory]
-    --previous-clustering PREVIOUS_CLUSTERING
-                          Directory containing previous cluster
-                          definitions and network [default = use that in
-                          the directory containing the model]
-    --previous-query-clustering PREVIOUS_QUERY_CLUSTERING
-                          Directory containing previous cluster
-                          definitions from poppunk_assign [default = use
-                          that in the directory containing the model]
-
-  Output options:
-    --output OUTPUT       Prefix for output files (required)
-    --overwrite           Overwrite any existing visualisation files
-
-  Database querying options:
-    --core-only           (with a 'refine' model) Use a core-distance
-                          only model for assigning queries [default =
-                          False]
-    --accessory-only      (with a 'refine' or 'lineage' model) Use an
-                          accessory-distance only model for assigning
-                          queries [default = False]
-
-  Visualisation options:
-    --microreact          Generate output files for microreact
-                          visualisation
-    --cytoscape           Generate network output files for Cytoscape
-    --phandango           Generate phylogeny and TSV for Phandango
-                          visualisation
-    --grapetree           Generate phylogeny and CSV for grapetree
-                          visualisation
-    --tree {nj,mst,both}  Type of tree to calculate [default = nj]
-    --mst-distances {core,accessory,euclidean}
-                          Distances used to calculate a minimum spanning
-                          tree [default = core]
-    --rapidnj RAPIDNJ     Path to rapidNJ binary to build NJ tree for
-                          Microreact
-    --perplexity PERPLEXITY
-                          Perplexity used to calculate t-SNE projection
-                          (with --microreact) [default=20.0]
-    --info-csv INFO_CSV   Epidemiological information CSV formatted for
-                          microreact (can be used with other outputs)
-
-  Other options:
-    --threads THREADS     Number of threads to use [default = 1]
-    --gpu-dist            Use a GPU when calculating distances [default
-                          = False]
-    --deviceid DEVICEID   CUDA device ID, if using GPU [default = 0]
-    --strand-preserved    If distances being calculated, treat strand as
-                          known when calculating random match chances
-                          [default = False]
-    --version             show program's version number and exit
diff --git a/docs/best_practises.rst b/docs/overview.rst
similarity index 73%
rename from docs/best_practises.rst
rename to docs/overview.rst
index 593bab24..86dadf91 100644
--- a/docs/best_practises.rst
+++ b/docs/overview.rst
@@ -1,31 +1,22 @@
-Best practises guide
+Overview
 ====================
 This page details the way in which we would advise that you *should* use and
 run PopPUNK, if possible.
 
+.. contents::
+   :local:
+
 .. image:: images/poppunk_flowchart.png
    :alt:  Flowchart for choosing how to use PopPUNK
    :align: center
 
-Use an online interface
------------------------
-If available, you may want to use one of the browser-based interfaces to
-PopPUNK. These include `PopPUNK-web <https://web.poppunk.net/>`__ and
-`pathogen.watch <https://pathogen.watch/genomes/all?genusId=1301&speciesId=1313>`__
-(*S. pneumoniae* only). See the :doc:`online` page for full details.
-
-Using these interfaces requires nothing to be installed or set up, doesn't require any
-genome data to be shared with us, and will return interactive visualisations. If your
-species isn't available, or you have large batches of genomes to cluster you will
-likely want to use the command line interface instead.
-
 Use the command line interface
 ------------------------------
 
 Installation and version
 ^^^^^^^^^^^^^^^^^^^^^^^^
-Install via conda if possible. Please use at least version v2.3.0 of PopPUNK
-and v1.5.1 of ``pp-sketchlib``.
+Install via conda if possible. Version 2.5.0 of PopPUNK and version 2.0.0 of pp-sketchlib
+are the current minimum supported versions.
 
 Use query assignment mode
 ^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -33,12 +24,11 @@ If a database is available for your species (see https://poppunk.net/pages/datab
 we would strongly recommend downloading it to use to cluster your genomes. This
 has many advantages:
 
-- No need to run through the potentially complex model fitting.
-- Assured model performance.
-- Considerable faster run times.
-- Use existing cluster definitions.
-- Use the context of large, high quality reference populations to interpret your
-  genomes' clusters.
+* No need to run through the potentially complex model fitting.
+* Assured model performance.
+* Considerably faster run times.
+* Use existing cluster definitions.
+* Use the context of large, high quality reference populations to interpret your genomes' clusters.
 
 See :doc:`query_assignment` for instructions on how to use this mode.
 
@@ -47,8 +37,12 @@ to define your sample's strains.
 
 Fit your own model
 ^^^^^^^^^^^^^^^^^^
-If a database isn't available for your species, you can fit your own. Details
-on how to do this can be found on :doc:`model_fitting`.
+If a database isn't available for your species, you can fit your own. This consists of three steps:
+
+1. Sketch your genomes (see :doc:`sketching`).
+2. Quality control your database (see :doc:`qc`).
+3. Fit a model (see :doc:`model_fitting`).
+4. Repeat step two, until you have a model which works for your needs.
 
 After getting a good fit, you may want to share it with others so that they can
 use it to assign queries. See :doc:`model_distribution` for advice. We would also
@@ -69,4 +63,20 @@ Older versions of PopPUNK mandated this be chosen as part of the main analysis,
 and then with ``--generate-viz`` mode. This is now run separately, after the
 main analysis, with ``poppunk_visualise``.
 
-See :doc:`visualisation` for details on options.
\ No newline at end of file
+See :doc:`visualisation` for details on options.
+
+Use an online interface
+-----------------------
+If available, you may want to use one of the browser-based interfaces to
+PopPUNK. These include `PopPUNK-web <https://web.poppunk.net/>`__ and
+`pathogen.watch <https://pathogen.watch/genomes/all?genusId=1301&speciesId=1313>`__
+(*S. pneumoniae* only).
+
+.. warning::
+   PopPUNK-web is temporarily down while we work on an updated and better supported
+   version.
+
+Using these interfaces requires nothing to be installed or set up, doesn't require any
+genome data to be shared with us, and will return interactive visualisations. If your
+species isn't available, or you have large batches of genomes to cluster you will
+likely want to use the command line interface instead.
diff --git a/docs/qc.rst b/docs/qc.rst
index 97541c24..4f019378 100644
--- a/docs/qc.rst
+++ b/docs/qc.rst
@@ -1,27 +1,39 @@
-Data quality control
-====================
-PopPUNK now comes with some basic quality control options which are applied by
-default when running ``--create-db``:
+Data quality control (``--qc-db``)
+==================================
+PopPUNK now comes with some basic quality control options, which you should
+run on your sketch database made with ``--create-db`` by running ``--qc-db``
+as follows::
 
-- Outlying genome length (calculated during sketching, for assemblies or reads).
-- Too many 'N's.
-- Outlying accessory distance.
+    poppunk --qc-db --ref-db example_db --type-isolate 12754_4_79 --length-range 2000000 3000000
 
-Accessory distance is also used with ``poppunk_assign``.
+For ``poppunk_assign``, instead add ``--run-qc``::
 
-You can set the behaviour when creating the database by setting ``--qc-filter``
-to one of the following three values if any failing samples are found:
+    poppunk_assign --query queries.txt --db example_db --run-qc --max-zero-dist 1 --max-merge 3
 
-- ``stop`` (default) -- Do not proceed past the sketching step, and throw an error.
-- ``prune`` -- Remove failing samples from the sketch database before continuing.
-- ``continue`` -- Ignore failing samples and run anyway.
+The following criteria are available:
+
+- Outlying genome length (calculated during sketching, for assemblies or reads) with ``--length-range`` and/or ``--length-sigma``.
+- Too many 'N's with ``--prop-n`` and/or ``--upper-n``.
+- Outlying core or accessory distances with ``--max-pi-dist`` and ``--max-a-dist`` respectively.
+- Too many zero distances with ``--max-zero-dist``.
+
+For ``poppunk --create-db`` only:
+
+- Names of samples to remove (e.g. failing external QC) with ``--remove-samples``.
+
+For ``poppunk_assign`` only:
+
+- Maximum number of clusters a single isolate can cause to merge with ``--max-merge``.
+- Betweenness of queries (not automated, just reported) with ``--betweenness``.
 
 In all cases a file will be written at ``qcreport.txt`` which lists the failing samples, and the
-reasons why they failed. If running with either prune or continue, you may also add ``--retain-failures``
+reasons why they failed. Adding ``--qc-keep`` will
+only write the file and not remove failing samples.
+You may also add ``--retain-failures``
 to write a separate sketch database with the failed samples.
 
 Random match chances in PopPUNK are only calculated and added to the database after the chosen
-QC step. If you use ``poppunk_sketch`` directly, they will be added without any automated QC.
+QC step. If you use ``sketchlib`` directly, they will be added without any automated QC.
 
 QC of input sequences
 ---------------------
@@ -38,7 +50,8 @@ and ``--upper-n`` which gives the absolute maximum value.
 QC of pairwise distances
 ------------------------
 The second QC step uses the pairwise distances, to enable the removal of outlier samples
-that may not be part of the taxon being studied.
+that may not be part of the taxon being studied. This is with reference to a type
+isolate. The type isolate will be selected by PopPUNK, unless specified using ``--type-isolate``.
 
 By default, the maximum allowed accessory distance is 0.5 to ensure you check for contamination.
 However, many species do really have high accessory values above this range, in which case you
@@ -47,82 +60,21 @@ should increase the value of ``--max-a-dist``.
 The maximum allowed core distance is also 0.5, by default. This can be altered with ``--max-pi-dist``.
 
 All sequences differing from the type isolate by distances greater than either threshold will be
-identified by the analysis, with the behaviour determined by the ``--qc-filter`` option. The type
-isolate will be selected by PopPUNK, unless specified using ``--type-isolate``.
-
-Removing samples from an existing database
-------------------------------------------
-You can use the ``poppunk_prune`` command to remove samples from a database,
-for example those found to be of poor quality. Create a file
-``remove.txt`` with the names of the samples you wish to remove, one per line,
-and run::
-
-   poppunk_prune --remove remove.txt --distances strain_db/strain_db.dists --output pruned_db
-
-This will remove the samples from the ``strain_db.dists`` files. This can instead be done
-simultaneously as the model is fitted - problematic sequences can be pruned, and the model fitted
-to the remaining high-quality samples by modifying the model fitting command to include QC options:
-
-  poppunk --fit-model dbscan --ref-db example_db --output example_dbscan --max-a-dist 0.4 --max-pi-dist 0.2
-
-Dealing with poor quality data
-------------------------------
-In this example we analyse 76 *Haemophilus influenzae* isolates. One isolate, 14412_4_15,
-is contaminated with 12% of reads being *Haemophilus parainfluenzae* and a total
-assembly length of 3.8Mb. It would be removed before input, but its presence
-can also be found with ``PopPUNK --qc-filter continue``.
-
-With the distances
-^^^^^^^^^^^^^^^^^^
-A fit with three mixture components overestimates the number of between strain
-links, and gives a network with a poor score (0.6849) and only five components:
-
-.. image:: images/contam_DPGMM_fit.png
-   :alt:  A bad fit to pairwise distances
-   :align: center
-
-Distances in the top left of the plot, with low core distances and high
-accessory distances, are due to the contaminated contigs in the isolate.
-Finding which isolates contribute to these distances reveals a clear culprit::
-
-    awk '$3<0.02 && $4 > 0.3 {print $1}' contam_db/contam_db.search.out | cut -f 1 | sort | uniq -c
-       1 14412_3_81
-       1 14412_3_82
-       1 14412_3_83
-       1 14412_3_84
-       1 14412_3_88
-       1 14412_3_89
-       1 14412_3_91
-       1 14412_3_92
-       1 14412_4_1
-       1 14412_4_10
-      28 14412_4_15
-
-In this case it is sufficient to increase the number of mixture components to four,
-which no longer includes these inflated distances. This gives a score of 0.9401 and 28 components:
-
-.. image:: images/contam_DPGMM_better_fit.png
-   :alt:  A better fit to pairwise distances
-   :align: center
-
-The best thing to do is to remove the poor quality isolate, or if possible
-remove the contaminated reads/contigs from the assembly.
-
-With the network
-^^^^^^^^^^^^^^^^
-Alternatively, the network itself can be inspected with ``--cytoscape``. Using
-the approach detailed in :ref:`cytoscape-view` gives the following view:
-
-.. image:: images/cytoscape_contaminant.png
-   :alt:  A better fit to pairwise distances
-   :align: center
-
-The contaminated node appears when ordering by ClusteringCoefficient, Stress or
-TopologicalCoefficient, and its edges appear when ordering by EdgeBetweeness.
-It can be seen highlighted in the top right component, connecting two clusters
-which otherwise have no links. It can be removed, and components recalculated in
-cytoscape directly, though removal from the PopPUNK database is best.
-
-The second largest cluster is also suspicious, where there are few triangles
-(low transitivity) and the nodes involved have high Stress. This is indicative
-of a bad fit overall, rather than a single problem sample.
+identified by the analysis.
+
+Each isolate may have a proportion of distances that are exactly zero as set by
+``--max-zero-dist``.
+
+QC of the network (assign only)
+-------------------------------
+Finally, you may also check network properties.
+
+Maximum number of clusters a single isolate can cause to merge is
+set with ``--max-merge``. More than this number of links across the original clusters
+will result in removal of the isolate.
+
+Betweenness of queries can be reported with ``--betweenness``, which may be useful
+to prune the input in more complex cases. This does not cause automated removal as
+it's difficult to set a sensible threshold across datasets.
+You will therefore need to re-run and remove samples yourself.
+
diff --git a/docs/query_assignment.rst b/docs/query_assignment.rst
index ca73dce1..6387c415 100644
--- a/docs/query_assignment.rst
+++ b/docs/query_assignment.rst
@@ -1,5 +1,5 @@
-Query assignment
-================
+Query assignment (``poppunk_assign``)
+=====================================
 This is the recommended mode to use PopPUNK, as long as a database is available for
 your species. If there is no database available, you can fit your own (:doc:`model_fitting`).
 
@@ -15,12 +15,6 @@ Downloading a database
 ----------------------
 Current PopPUNK databases can be found here: https://poppunk.net/pages/databases.html
 
-.. note::
-    We currently only have three reference databases available for public use.
-    Please bear with us as we update this, more reference databases which work
-    with version 2 of PopPUNK will be released shortly. You can still fit your
-    own model to your data.
-
 We refer to sequences in the database as references, and those being added
 as queries. The clusters assigned by PopPUNK are variable-length-k-mer clusters (VLKCs).
 
@@ -150,14 +144,6 @@ to all of these examples. This may give you a one-to-one mapping between PopPUNK
 and your external cluster, or you may find multiple external clusters refer to the
 PopPUNK cluster giving output such as ``227;811;763;824``.
 
-Using a model fitted with lineage assignment mode
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-You will need to add ``--assign-lineages`` to pick up the correct model, additionally
-you can add options ``--rank`` to choose the rank to assign from (default is the lowest
-available) and ``--use-accessory`` to use the accessory distances rather than the core
-for clustering. You will find extra model files with the ranks listed in their name if
-this model type is available.
-
 Using a model fitted with ``--indiv-refine``
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 If the database was fitted with the refine fit mode, and ``indiv-refine`` you may have
diff --git a/docs/scripts.rst b/docs/scripts.rst
index a53be0e8..f40e99c3 100644
--- a/docs/scripts.rst
+++ b/docs/scripts.rst
@@ -54,6 +54,8 @@ This script will do the following:
 
 5. Output the combined clusters, average core distances, and tree.
 
+6. Cut this tree to pick a set of clusters under a similarity given by ``--cutoff``.
+
 Adding weights to the network
 -----------------------------
 Converts binary within-cluster edge weights to the Euclidean core-accessory distance.
diff --git a/docs/setup.py b/docs/setup.py
index 8c3bb63b..0bdff49b 100644
--- a/docs/setup.py
+++ b/docs/setup.py
@@ -57,7 +57,7 @@ def find_version(*file_paths):
             'poppunk = PopPUNK.__main__:main',
             'poppunk_prune = PopPUNK.prune_db:main',
             'poppunk_references = PopPUNK.reference_pick:main',
-            'poppunk_tsne = PopPUNK.tsne:main'
+            'poppunk_mandrake = PopPUNK.mandrake:main'
             ]
     },
     scripts=['scripts/poppunk_calculate_rand_indices.py',
diff --git a/docs/sketching.rst b/docs/sketching.rst
index c3d12fdf..dac58f73 100644
--- a/docs/sketching.rst
+++ b/docs/sketching.rst
@@ -1,5 +1,5 @@
-Sketching
-=========
+Sketching (``--create-db``)
+===========================
 
 The basis of all analysis is estimation of core and accessory genome distances between samples.
 PopPUNK uses genome sketching to make analysis more efficient. In previous versions we used
@@ -52,15 +52,14 @@ You can use pp-sketchlib directly to create sketches, though functionality is id
 to doing this through PopPUNK. You will need to run both sketch and query modes to generate
 the sketch database and the distance files as in ``--create-db``::
 
-   poppunk_sketch --sketch --rfile rfiles.txt --ref-db database --sketch-size 10000 --min-k 15 --k-step 2 --cpus 4
-   poppunk_sketch --query --ref-db database --query-db database --cpus 4
+   sketchlib sketch -l rfiles.txt -o database -s 10000 -k 15,31,2 --cpus 4
+   sketchlib query dist database -o dists --cpus 4
 
 You may want to do this if you anticipate trying different k-mer sizes, are using the
 databases for other purposes, or running a very large analysis where it is useful to split
 up the sketching and distance steps. Useful options include:
 
-- ``--print`` -- to print distances in human-readable format to the terminal.
-- ``--jaccard`` -- will output Jaccard distances at each k-mer length, rather than core and accessory distances.
+- ``sketchlib query jaccard`` -- will output Jaccard distances at each k-mer length, rather than core and accessory distances.
 - ``--subset`` -- to only calculate distances for a subset of the genomes in the reference database.
 
 .. warning::
@@ -70,7 +69,7 @@ up the sketching and distance steps. Useful options include:
 
 Viewing information about a database
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-Use ``poppunk_db_info`` on a HDF5 file::
+Use ``poppunk_info`` on a HDF5 file::
 
     PopPUNK database:		ecoli.h5
     Sketch version:			9314bda28ed25a60dd40f9b9e896c0b269500fec
diff --git a/docs/subclustering.rst b/docs/subclustering.rst
index f8f2970c..a9bd5324 100644
--- a/docs/subclustering.rst
+++ b/docs/subclustering.rst
@@ -6,10 +6,6 @@ Subclustering with PopPIPE
 .. contents::
    :local:
 
-.. danger::
-    PopPIPE is still a work-in-progress and may not immediately work on your data. We
-    anticipate finalising the pipeline in the near future.
-
 Overview
 --------
 You can run `PopPIPE <https://github.com/johnlees/PopPIPE>`__ on your PopPUNK output,
@@ -44,7 +40,7 @@ Other dependencies will be automatically installed by conda the first time
 you run the pipeline. You can also install them yourself and omit the `-use-conda`
 directive to snakemake::
 
-    conda create -n poppipe --file=environment.yml
+    conda env create -n poppipe --file=environment.yml
 
 Then clone the repository::
 
diff --git a/docs/troubleshooting.rst b/docs/troubleshooting.rst
index f17825ae..a9582677 100644
--- a/docs/troubleshooting.rst
+++ b/docs/troubleshooting.rst
@@ -4,11 +4,87 @@ Troubleshooting
 This page deals with common issues in running the analysis. For issues with
 installing or running the software please raise an issue on github.
 
-.. contents::
-   :local:
+Most/all of my samples merge when I run a query
+-----------------------------------------------
+If you see a gigantic merge when running ``poppunk_assign``, for example::
+
+    Clusters 1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16 have merged into 1_2_3_4_5_6_7_8_9_10_11_12_13_14_15_16
+
+This might be caused by:
+
+* A single/a few 'bad' genome(s) with artificial zero/close-to-zero distances to many other samples, which links much of the network together.
+* A boundary which is actually a bit lax, and when more isolates are added,
+  they have distances just below the boundary between a few clusters,
+  and there are enough of these to link more and more clusters together.
+
+The first issue should be easy to fix with one of the following methods:
+
+* Use ``--run-qc`` (see :doc:`qc`) with one or more of:
+
+    * ``--max-pi-dist`` and/or ``--max-a-dist`` to remove outlier distances.
+    * ``--max-zero-dist`` which sets the maximum proportion of zeros to other samples. Poor quality genomes often end up with zero distances to many samples, linking them together.
+    * ``--max-merge`` which sets the maximum number of clusters a single query can cause to merge.
+    * ``--betweenness`` which will print all queries in order which have high stress in the network, and are likely causing merges.
+* Use ``--serial`` to run samples through one-by-one. This is a little less
+  efficient than fully batched querying, but much faster than running independent
+  jobs. Note, lineage models and updating the database are not supported with
+  this mode.
+
+The second issue above is potentially more insidiuous, and may require a refit to all the
+data to obtain a tighter boundary. You can (mostly) keep old cluster IDs via
+the use of ``--external-clustering`` if you do this.
+Alternatively, you can add the ``--serial`` command to type samples one at a time
+as above.
+
+See `issue 194 <https://github.com/bacpop/PopPUNK/issues/194>`__ for more discussion.
+
+Memory/run-time issues
+----------------------
+Here are some tips based on experiences analysing larger datasets:
+
+- Add ``--threads`` -- they are used fairly efficiently throughout.
+- Consider the ``--gpu-sketch`` and ``--gpu-dists`` options is applicable,
+  and a GPU is available.
+- In ``--refine-model`` set ``--pos-shift 0`` to avoid creating huge networks
+  with close to :math:`N^2` edges. Mixture models normally need to be pruned.
+- In ``--refine-model`` you may add the ``--no-local`` option to skip that step
+  and decrease run-time, though gains are likely marginal.
+- Use ``--rapid-nj``, if producing microreact output.
+
+Another option for scaling is to run ``--create-db`` with a smaller initial set (not
+using the ``--full-db`` command), then use ``--assign-query`` to add to this.
 
 Known bugs
 ----------
+Calculating query-query distances when unnecessary
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+A bug in v2.4.0 only, (fixed in v2.5.0 and not in previous versions).
+
+You will always see ``Found novel query clusters. Calculating distances between them.``
+when running ``poppunk_assign`` with more than one input sample. This should only
+happen when unassigned isolates/novel clusters are found. Our check on this condition
+became invalid.
+
+Additionally, this *may* have affected typing when query-query links were present,
+this appeared as invalid merges in some tests. If you used this, you may wish to
+re-run with v2.5.0 or higher.
+
+Older HDBSCAN models fail to load
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+tl;dr if you see an error ``ModuleNotFoundError: No module named 'sklearn.neighbors._dist_metrics'``
+you probably need to downgrade ``sklearn`` to v0.24.
+
+The change in scikit-learn's API in v1.0.0 and above mean that HDBSCAN models
+fitted with ```sklearn <=v0.24``` will give an error when loaded. If you run into this,
+the solution is one of:
+- Downgrade sklearn to v0.24.
+- Run model refinement to turn your model into a boundary model instead (this will
+change clusters).
+- Refit your model in an environment with ```sklearn >=v1.0``.
+
+If this is a common problem let us know, as we could write a script to 'upgrade'
+HDBSCAN models.
+See issue [#213](https://github.com/bacpop/PopPUNK/issues/213) for more details.
 
 When I look at my clusters on a tree, they make no sense
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -79,18 +155,4 @@ used to assign new queries.
 If you want to change cluster names or assign queries to your own cluster definitions
 you can use the ``--external-clustering`` argument instead.
 
-Memory/run-time issues
-----------------------
-Here are some tips based on experiences analysing larger datasets:
 
-- Add ``--threads`` -- they are used fairly efficiently throughout.
-- Consider the ``--gpu-sketch`` and ``--gpu-dists`` options is applicable,
-  and a GPU is available.
-- In ``--refine-model`` set ``--pos-shift 0`` to avoid creating huge networks
-  with close to :math:`N^2` edges. Mixture models normally need to be pruned.
-- In ``--refine-model`` you may add the ``--no-local`` option to skip that step
-  and decrease run-time, though gains are likely marginal.
-- Use ``--rapid-nj``, if producing microreact output.
-
-Another option for scaling is to run ``--create-db`` with a smaller initial set (not
-using the ``--full-db`` command), then use ``--assign-query`` to add to this.
diff --git a/docs/visualisation.rst b/docs/visualisation.rst
index 43d655e8..5db6d4c7 100644
--- a/docs/visualisation.rst
+++ b/docs/visualisation.rst
@@ -73,28 +73,42 @@ Adding the ``--microreact`` flag will create the following files:
 
 - _microreact_clusters.csv -- the strain or lineage assignments with headers formatted for microreact, plus anything from ``--info-csv``.
 - _core_NJ.nwk -- a neighbour-joining tree from the core distances.
-- _perplexity20.0_accessory_tsne.dot -- a 2D embedding of the accessory distances, produced using t-SNE (in this case with
+- _perplexity20.0_accessory_mandrake.dot -- a 2D embedding of the accessory distances, produced using `mandrake <https://github.com/bacpop/mandrake>`__ (in this case with
   :ref:`perplexity` 20).
 
-Open https://microreact.org/upload in your browser, and drag and drop these three files
+From version 2.5.0 this will also include:
+
+- .microreact -- a Microreact compatible JSON containing all of the above, which
+   can be uploaded directly.
+
+If you add ``--api-key`` and provide `your account's API key <https://docs.microreact.org/api/access-tokens>__`
+this will automatically create an instance, and the URL will be output to the terminal.
+
+Otherwise, open https://microreact.org/upload in your browser, and drag and drop these three files
 to create your visualisation. Here is the result of running the visualisation on the
 *Listeria* BGMM model::
 
-   poppunk_visualise --ref-db listeria --microreact
+    poppunk_visualise --ref-db listeria --microreact --threads 8
 
-   Graph-tools OpenMP parallelisation enabled: with 1 threads
-   PopPUNK: visualise
-   Loading BGMM 2D Gaussian model
-   Writing microreact output
-   Parsed data, now writing to CSV
-   Building phylogeny
-   Running t-SNE
+    Graph-tools OpenMP parallelisation enabled: with 8 threads
+    PopPUNK: visualise
+    Loading BGMM 2D Gaussian model
+    Completed model loading
+    Building phylogeny
+    Writing microreact output
+    Parsed data, now writing to CSV
+    Running mandrake
+    Running on CPU
+    Preprocessing 128 samples with perplexity = 20 took 0ms
+    Optimizing	 Progress: 99.9%, eta=0.0010, Eq=0.2583546852, clashes=2.1%
+    Optimizing done in 30s
+    Provide --api-key to create microreact automatically
 
-   Done
+    Done
 
-This can be viewed at https://microreact.org/project/8PeGg9fCjZADaAGuNJwU9z:
+This can be viewed at https://microreact.org/project/3JAZKqzJiaNyViWXindNLv-listeria-poppunk-perplexity-20:
 
-.. image:: images/listeria_microreact.png
+.. image:: images/microreact.png
    :alt:  Microreact page for Listeria monocytogenes
    :align: center
 
@@ -105,43 +119,40 @@ to compare results from different ranks.
 
 .. _perplexity:
 
-Setting the perplexity parameter for t-SNE
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-In t-SNE an embedding of the accessory genome distances is found which
+Setting the perplexity parameter for mandrake
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+In mandrake an embedding of the accessory genome distances is found which
 represents local structure of the data. Isolates with similar accessory content
 will visually appear in clusters together.
 
 The perplexity sets a guess about the number of close neighbours each point
-has, and is a trade-off between local and global structure. t-SNE is reasonably
+has, and is a trade-off between local and global structure. t-SNE (and by extension mandrake) is reasonably
 robust to changes in the perplexity parameter (set with ``--perplexity`` when
 creating microreact output with ``--microreact``),
 however we would recommend trying a few values to get
 a good embedding for the accessory distances.
 
-There is a good discussion of the effect of perplexity `here <https://distill.pub/2016/misread-tsne/>`_
+There is a good discussion of the effect of perplexity `here <https://distill.pub/2016/misread-tsne/>`__
 and the sklearn documentation shows some examples of the effect of `changing
-perplexity <http://scikit-learn.org/stable/auto_examples/manifold/plot_t_sne_perplexity.html>`_.
+perplexity <http://scikit-learn.org/stable/auto_examples/manifold/plot_t_sne_perplexity.html>`__.
+In mandrake, points will usually appear 'tighter' than in t-SNE, and form more obvious clusters.
 
-In the :doc:`model_fitting` example, a perplexity of 30 gives clear clustering of
-the accessory genome content, condordant with the core genome structure (`data <https://microreact.org/project/Skg0j9sjz>`__):
-
-.. image:: images/microreact.png
-   :alt:  Microreact plot of results with perplexity = 30
-   :align: center
+In the example with *Listeria monocytogenes* above, a perplexity of 20 gives clear clustering of
+the accessory genome content, condordant with the core genome structure (`data <https://microreact.org/project/3JAZKqzJiaNyViWXindNLv-listeria-poppunk-perplexity-20>`__):
 
-With a lower perplexity of 5, the clustering is too loose, and the strain
-structure cannot clearly be seen (`data <https://microreact.org/project/S1RwpK9if>`__):
+With a lower perplexity of 5, the clustering is not as tight, but it still looks ok
+(`data <https://microreact.org/project/tXHmDR4NRfmTeemfjqbzip-listeria-poppunk-perplexity-5>`__):
 
 .. image:: images/microreact_perplexity5.png
    :alt:  Microreact plot of results with perplexity = 5
    :align: center
 
 30 is a good default, but you may wish to try other values, particularly with
-larger or smaller datasets. You can re-run the t-SNE using the ``poppunk_tsne``
+larger or smaller datasets. You can re-run the mandrake using the ``poppunk_mandrake``
 command, providing the distances from the previous run::
 
-   poppunk_tsne --distances strain_db/strain_db.dists --output strain_db \
-   --perplexity 20 --verbosity 1
+   poppunk_mandrake --distances strain_db/strain_db.dists --output strain_db \
+   --perplexity 50
 
 GrapeTree
 ---------
@@ -197,9 +208,10 @@ Cytoscape is different from the above modes as it creates a layout and visualisa
 the graph used to create strains from distances. This can be useful for more detailed
 investigation of network scores, particularly in strains which have less than perfect transitivity.
 
-Add the ``--cytoscape`` option::
+Add the ``--cytoscape`` option, and also ``--network-file`` to point to the
+network you wish to visualise::
 
-   poppunk_visualise --ref-db listeria --cytoscape
+   poppunk_visualise --ref-db listeria --cytoscape --network-file listeria/listeria_graph.gt
 
    Graph-tools OpenMP parallelisation enabled: with 1 threads
    PopPUNK: visualise
diff --git a/environment.yml b/environment.yml
index 6b5b4e27..e6831eb3 100644
--- a/environment.yml
+++ b/environment.yml
@@ -6,24 +6,22 @@ channels:
   - r
 dependencies:
   - pip
-  - numpy
+  - numpy <=1.22.4
   - scipy
   - python-dateutil
   - pandas
   - scikit-learn >=0.24
+  - mandrake
   - matplotlib-base
-  - dendropy >=4.4.0
+  - biopython
   - treeswift
   - matplotlib
   - hdbscan
   - rapidnj
   - h5py
-  - pp-sketchlib >=1.7.0
-  - graph-tool >=2.35
   - requests
-  - flask
-  - flask-cors
-  - gunicorn
+  - pp-sketchlib >=2.0.0
+  - graph-tool >=2.35
   - networkx
   - xorg-libxi
   - xorg-libxcursor
@@ -42,5 +40,3 @@ dependencies:
   - boost-cpp
   - libgomp
   - tqdm
-  - tzlocal <3.0
-  - flask-apscheduler
diff --git a/poppunk_logo.png b/poppunk_logo.png
index f29e7590..79517840 100644
Binary files a/poppunk_logo.png and b/poppunk_logo.png differ
diff --git a/poppunk_mandrake-runner.py b/poppunk_mandrake-runner.py
new file mode 100755
index 00000000..7728a4d2
--- /dev/null
+++ b/poppunk_mandrake-runner.py
@@ -0,0 +1,10 @@
+#!/usr/bin/env python
+# vim: set fileencoding=<utf-8> :
+# Copyright 2018-2022 John Lees and Nick Croucher
+
+"""Convenience wrapper for running poppunk_mandrake directly from source tree."""
+
+from PopPUNK.mandrake import main
+
+if __name__ == '__main__':
+    main()
diff --git a/poppunk_prune-runner.py b/poppunk_prune-runner.py
deleted file mode 100755
index f4ca2bc2..00000000
--- a/poppunk_prune-runner.py
+++ /dev/null
@@ -1,10 +0,0 @@
-#!/usr/bin/env python
-# vim: set fileencoding=<utf-8> :
-# Copyright 2018-2020 John Lees and Nick Croucher
-
-"""Convenience wrapper for running prune_poppunk directly from source tree."""
-
-from PopPUNK.prune_db import main
-
-if __name__ == '__main__':
-    main()
diff --git a/poppunk_tsne-runner.py b/poppunk_tsne-runner.py
deleted file mode 100755
index 0321f8a5..00000000
--- a/poppunk_tsne-runner.py
+++ /dev/null
@@ -1,10 +0,0 @@
-#!/usr/bin/env python
-# vim: set fileencoding=<utf-8> :
-# Copyright 2018-2020 John Lees and Nick Croucher
-
-"""Convenience wrapper for running poppunk_tsne directly from source tree."""
-
-from PopPUNK.tsne import main
-
-if __name__ == '__main__':
-    main()
diff --git a/pyproject.toml b/pyproject.toml
new file mode 100644
index 00000000..20bf631f
--- /dev/null
+++ b/pyproject.toml
@@ -0,0 +1,3 @@
+[build-system]
+requires = ["setuptools>=42", "wheel", "pybind11~=2.6.1"]
+build-backend = "setuptools.build_meta"
diff --git a/requirements.txt b/requirements.txt
index 4bfc4281..65392a9f 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -1,1 +1,19 @@
-Cython>=0.26.1
\ No newline at end of file
+Cython>=0.26.1
+pybind11
+dendropy>=4.4.0
+flask
+flask-cors
+gunicorn
+h5py
+hdbscan
+matplotlib
+networkx
+numpy
+pandas
+pp-sketchlib>=1.7.0
+requests
+scikit-learn>=0.24
+scipy
+tqdm
+treeswift
+tzlocal<3.0
diff --git a/scripts/poppunk_iterate.py b/scripts/poppunk_iterate.py
index b2cc21fc..558bf237 100644
--- a/scripts/poppunk_iterate.py
+++ b/scripts/poppunk_iterate.py
@@ -1,6 +1,6 @@
 #!/usr/bin/env python
 # vim: set fileencoding=<utf-8> :
-# Copyright 2021 John Lees, Nick Croucher and Bin Zhao
+# Copyright 2021-2022 John Lees, Nick Croucher and Bin Zhao
 
 from collections import defaultdict
 import sys
@@ -14,6 +14,33 @@
 
 import pp_sketchlib
 
+# https://stackoverflow.com/a/17954769
+from contextlib import contextmanager
+@contextmanager
+def stderr_redirected(to=os.devnull):
+    '''
+    import os
+
+    with stdout_redirected(to=filename):
+        print("from Python")
+        os.system("echo non-Python applications are also supported")
+    '''
+    fd = sys.stderr.fileno()
+
+    def _redirect_stderr(to):
+        sys.stderr.close()
+        os.dup2(to.fileno(), fd)
+        sys.stderr = os.fdopen(fd, 'w')
+
+    with os.fdopen(os.dup(fd), 'w') as old_stderr:
+        with open(to, 'w') as file:
+            _redirect_stderr(to=file)
+        try:
+            yield
+        finally:
+            _redirect_stderr(to=old_stderr)
+
+
 # command line parsing
 def get_options():
 
@@ -35,6 +62,12 @@ def get_options():
         default=None,
         help="Prefix for output files [default = <--db>/<--db>_iterate",
     )
+    parser.add_argument(
+        "--cutoff",
+        default=0.1,
+        type=float,
+        help="Proportional distance cutoff below which to generate clusters, between 0 and 1 [default = 0.1]",
+    )
     parser.add_argument(
         "--cpus", default=1, type=int, help="Number of CPUs to use [default = 1]"
     )
@@ -108,10 +141,12 @@ def is_nested(cluster_dict, child_members, node_list):
 
     # Check input ok
     args = get_options()
+    if args.cutoff >= 1 or args.cutoff <= 0:
+        raise RuntimeError("--cutoff must be between 0 and 1\n")
     if args.output is None:
-        args.output = args.db + "/" + args.db + "_iterate"
+        args.output = f"{args.db}/{args.db}_iterate"
     if args.h5 is None:
-        args.h5 = args.db + "/" + args.db
+        args.h5 = f"{args.db}/{args.db}"
     else:
         # Remove the .h5 suffix if present
         h5_prefix = re.match("^(.+)\.h5$", args.h5)
@@ -119,7 +154,7 @@ def is_nested(cluster_dict, child_members, node_list):
             args.h5 = h5_prefix.group(1)
 
     # Set up reading clusters
-    db_name = args.db + "/" + os.path.basename(args.db)
+    db_name = f"{args.db}/{os.path.basename(args.db)}"
     cluster_it = read_next_cluster_file(db_name)
     all_clusters, iterated_clusters, first_idx = next(cluster_it)
     all_samples = set()
@@ -148,9 +183,9 @@ def is_nested(cluster_dict, child_members, node_list):
 
     # Calculate core distances
     # Set up sketchlib args
-    ref_db_handle = h5py.File(args.h5 + ".h5", "r")
+    ref_db_handle = h5py.File(f"{args.h5}.h5", "r")
     first_sample_name = list(ref_db_handle["sketches"].keys())[0]
-    kmers = ref_db_handle["sketches/" + first_sample_name].attrs["kmers"]
+    kmers = ref_db_handle[f"sketches/{first_sample_name}"].attrs["kmers"]
     kmers = np.asarray(sorted(kmers))
     random_correct = True
     jaccard = False
@@ -159,21 +194,25 @@ def is_nested(cluster_dict, child_members, node_list):
 
     # Run a query for each cluster
     pi_values = {}
-    for cluster in sorted_clusters:
-        rNames = list(iterated_clusters[cluster])
-        distMat = pp_sketchlib.queryDatabase(
-            args.h5,
-            args.h5,
-            rNames,
-            rNames,
-            kmers,
-            random_correct,
-            jaccard,
-            args.cpus,
-            use_gpu,
-            deviceid,
-        )
-        pi_values[cluster] = np.mean(distMat[:, 0])
+    max_pi = -1.0
+    # Hide the progress bars
+    with stderr_redirected():
+        for cluster in sorted_clusters:
+            rNames = list(iterated_clusters[cluster])
+            distMat = pp_sketchlib.queryDatabase(
+                args.h5,
+                args.h5,
+                rNames,
+                rNames,
+                kmers,
+                random_correct,
+                jaccard,
+                args.cpus,
+                use_gpu,
+                deviceid,
+            )
+            pi_values[cluster] = np.mean(distMat[:, 0])
+            max_pi = max(max_pi, pi_values[cluster])
 
     # Nest the clusters
     tree = Tree()
@@ -182,7 +221,7 @@ def is_nested(cluster_dict, child_members, node_list):
 
     tree_clusters = deepcopy(iterated_clusters)
     node_list = {"root": root_node}
-    tree_clusters["root"] = all_samples
+    tree_clusters["root"] = all_samples.copy()
     for cluster in sorted_clusters:
         new_node = Node(label="cluster" + str(cluster))
         parent_cluster = is_nested(
@@ -202,17 +241,61 @@ def is_nested(cluster_dict, child_members, node_list):
             node_list[cluster].add_child(Node(label=sample))
 
     # Write output
-    tree.write_tree_newick(args.output + ".tree.nwk", hide_rooted_prefix=True)
-    with open(args.output + ".clusters.csv", "w") as f:
+    tree.write_tree_newick(f"{args.output}.tree.nwk", hide_rooted_prefix=True)
+    with open(f"{args.output}.clusters.csv", "w") as f:
         f.write("Cluster,Avg_Pi,Taxa\n")
         for cluster in sorted_clusters:
-            f.write(
-                str(cluster)
-                + ","
-                + str(pi_values[cluster])
-                + ","
-                + ";".join(iterated_clusters[cluster])
-                + "\n"
-            )
+            f.write(f"{str(cluster)},{str(pi_values[cluster])},{';'.join(iterated_clusters[cluster])}\n")
+
+    # Now add lengths in for the cut algorithm
+    for node in tree.traverse_preorder(leaves=False):
+        cluster_label = re.match(r"^cluster(\d+)$", node.get_label())
+        if cluster_label and cluster_label.group(1):
+            node.set_edge_length(pi_values[int(cluster_label.group(1))])
+        elif node.get_label() != "root":
+            raise RuntimeError(f"Couldn't parse cluster {node.get_label()}")
+    tree.scale_edges(1 / max_pi)
+
+    cut_clusters = set()
+    for leaf in tree.traverse_leaves():
+        parent_node = leaf.get_parent()
+        if not parent_node.is_root():
+            # For each leaf, go back up the tree to find the cluster which
+            # crosses the cutoff threshold
+            while not parent_node.is_root() and\
+                parent_node.get_edge_length() < args.cutoff:
+                # Go up a level
+                child_node = parent_node
+                parent_node = child_node.get_parent()
+                if parent_node.is_root() or \
+                    (child_node.get_edge_length() < args.cutoff and \
+                     parent_node.get_edge_length() > args.cutoff):
+                    cut_clusters.add(parent_node)
+
+    # In the case where a grand-parent node's average core distance is smaller
+    # than child node's both great parent node and child node will be selected
+    # Remove any ancestors to fix this
+    unwanted_parents = set()
+    for selected_node in cut_clusters:
+        for parent in selected_node.traverse_ancestors(include_self=False):
+            if parent.get_label() in cut_clusters:
+                unwanted_parents.add(parent.get_label())
+                break
+    cut_clusters -= unwanted_parents
+
+    included_samples = set()
+    with open(f"{args.output}.cutoff_clusters.csv", "w") as f:
+        f.write("Isolate,Cluster\n")
+        for idx, selected_node in enumerate(cut_clusters):
+            cluster_label = re.match(r"^cluster(\d+)$", selected_node.get_label())
+            if cluster_label and cluster_label.group(1):
+                for sample in iterated_clusters[int(cluster_label.group(1))]:
+                    included_samples.add(sample)
+                    f.write(f"{sample},{idx + 1}\n")
+            elif selected_node.get_label() != "root":
+                raise RuntimeError(f"Couldn't parse cluster {selected_node.get_label()}")
+        singletons = all_samples - included_samples
+        for idx, sample in enumerate(singletons):
+            f.write(f"{sample},{idx + len(cut_clusters) + 1}\n")
 
     sys.exit(0)
diff --git a/scripts/poppunk_pickle_fix.py b/scripts/poppunk_pickle_fix.py
deleted file mode 100755
index bfde518f..00000000
--- a/scripts/poppunk_pickle_fix.py
+++ /dev/null
@@ -1,36 +0,0 @@
-#!/usr/bin/env python
-# vim: set fileencoding=<utf-8> :
-# Copyright 2018-2020 John Lees and Nick Croucher
-
-import sys
-import argparse
-import pickle
-
-# command line parsing
-def get_options():
-
-    parser = argparse.ArgumentParser(description='Fix pickle files with incorrect label order',
-                                     prog='pickle_fix')
-
-    # input options
-    parser.add_argument('pickle', help='Input pickle (.dists.pkl)')
-    parser.add_argument('output', help='Prefix for output files')
-
-    return parser.parse_args()
-
-# main code
-if __name__ == "__main__":
-
-    # Check input ok
-    args = get_options()
-
-    with open(args.pickle, 'rb') as pickled_names:
-        rNames, qNames, self = pickle.load(pickled_names)
-
-    rNames = sorted(rNames)
-    qNames = sorted(qNames)
-
-    with open(args.output + ".dists.pkl", 'wb') as pickle_fixed:
-        pickle.dump([rNames, qNames, self], pickle_fixed)
-
-    sys.exit(0)
diff --git a/setup.py b/setup.py
old mode 100644
new mode 100755
index f74b973b..2519ef45
--- a/setup.py
+++ b/setup.py
@@ -92,7 +92,7 @@ def build_extension(self, ext):
     long_description=long_description,
     long_description_content_type='text/markdown',
     url='https://github.com/johnlees/PopPUNK',
-    author='John Lees and Nicholas Croucher',
+    author='John Lees, Nicholas Croucher, Marie Gronemeyer, Richard Fitzjohn',
     author_email='john@johnlees.me',
     license='Apache Software License',
     classifiers=[
@@ -111,9 +111,8 @@ def build_extension(self, ext):
             'poppunk_assign = PopPUNK.assign:main',
             'poppunk_visualise = PopPUNK.visualise:main',
             'poppunk_mst = PopPUNK.sparse_mst:main',
-            'poppunk_prune = PopPUNK.prune_db:main',
             'poppunk_references = PopPUNK.reference_pick:main',
-            'poppunk_tsne = PopPUNK.tsne:main',
+            'poppunk_mandrake = PopPUNK.mandrake:main',
             'poppunk_info = PopPUNK.info:main'
             ]
     },
@@ -124,12 +123,12 @@ def build_extension(self, ext):
              'scripts/poppunk_extract_distances.py',
              'scripts/poppunk_iterate.py',
              'scripts/poppunk_add_weights.py',
-             'scripts/poppunk_easy_run.py',
-             'scripts/poppunk_pickle_fix.py'],
+             'scripts/poppunk_easy_run.py'],
     ext_modules=[CMakeExtension('poppunk_refine')],
     test_suite="test",
     cmdclass=dict(build_ext=CMakeBuild),
     zip_safe=False,
     include_package_data=True,
-    package_data={'': ['PopPUNK/data/*.json.gz']}
+    package_data={'': ['PopPUNK/data/*.json.gz',
+                       'PopPUNK/data/microreact_example.pkl']}
 )
diff --git a/src/boundary.hpp b/src/boundary.hpp
index 6aff8162..f188920d 100644
--- a/src/boundary.hpp
+++ b/src/boundary.hpp
@@ -16,7 +16,7 @@
 #include <Eigen/Dense>
 
 // Parallel sort
-#include <boost/sort/sort.hpp>
+#include <boost/sort/parallel_stable_sort/parallel_stable_sort.hpp>
 typedef Eigen::Matrix<float, Eigen::Dynamic, Eigen::Dynamic, Eigen::RowMajor>
     NumpyMatrix;
 typedef std::tuple<std::vector<long>, std::vector<long>, std::vector<long>>
diff --git a/vendor/boost/sort/common/indirect.hpp b/vendor/boost/sort/common/indirect.hpp
new file mode 100644
index 00000000..a55ef820
--- /dev/null
+++ b/vendor/boost/sort/common/indirect.hpp
@@ -0,0 +1,153 @@
+//----------------------------------------------------------------------------
+/// @file indirect.hpp
+/// @brief Indirect algorithm
+///
+/// @author Copyright (c) 2016 Francisco Jose Tapia (fjtapia@gmail.com )\n
+///         Distributed under the Boost Software License, Version 1.0.\n
+///         ( See accompanying file LICENSE_1_0.txt or copy at
+///           http://www.boost.org/LICENSE_1_0.txt  )
+/// @version 0.1
+///
+/// @remarks
+//-----------------------------------------------------------------------------
+#ifndef __BOOST_SORT_PARALLEL_COMMON_INDIRECT_HPP
+#define __BOOST_SORT_PARALLEL_COMMON_INDIRECT_HPP
+
+//#include <boost/sort/common/atomic.hpp>
+#include <boost/sort/common/util/traits.hpp>
+#include <functional>
+#include <iterator>
+#include <type_traits>
+#include <vector>
+
+namespace boost
+{
+namespace sort
+{
+namespace common
+{
+
+//
+//---------------------------------------------------------------------------
+/// @struct less_ptr_no_null
+///
+/// @remarks this is the comparison object for pointers. Compare the objects
+///          pointed by the iterators
+//---------------------------------------------------------------------------
+template<class Iter_t, class Compare = util::compare_iter<Iter_t> >
+struct less_ptr_no_null
+{
+    //----------------------------- Variables -----------------------
+    Compare comp; // comparison object of the elements pointed by Iter_t
+
+    //------------------------------------------------------------------------
+    //  function : less_ptr_no_null
+    /// @brief constructor from a Compare object
+    /// @param C1 : comparison object
+    //-----------------------------------------------------------------------
+    less_ptr_no_null(Compare C1 = Compare()): comp(C1) { };
+
+    //------------------------------------------------------------------------
+    //  function : operator ( )
+    /// @brief Make the comparison of the objects pointed by T1 and T2, using
+    //         the internal comp
+    //
+    /// @param  T1 : first iterator
+    /// @param  T2 : second iterator
+    /// @return bool result of the comparison
+    //-----------------------------------------------------------------------
+    bool operator( )(Iter_t T1, Iter_t T2) const
+    {
+        return comp(*T1, *T2);
+    };
+};
+//
+//-----------------------------------------------------------------------------
+//  function : create_index
+/// @brief From a vector of objects, create a vector of iterators to
+///        the objects
+///
+/// @param first : iterator to the first element of the range
+/// @param last : iterator to the element after the last of the range
+/// @param index : vector where store the iterators
+//-----------------------------------------------------------------------------
+template<class Iter_t>
+static void create_index(Iter_t first, Iter_t last, std::vector<Iter_t> &index)
+{
+    auto nelem = last - first;
+    assert(nelem >= 0);
+    index.clear();
+    index.reserve(nelem);
+    for (; first != last; ++first) index.push_back(first);
+};
+//
+//-----------------------------------------------------------------------------
+//  function : sort_index
+/// @brief This function transform a logical sort of the elements in the index
+///        in a physical sort
+//
+/// @param global_first : iterator to the first element of the data
+/// @param [in] index : vector of the iterators
+//-----------------------------------------------------------------------------
+template<class Iter_t>
+static void sort_index(Iter_t global_first, std::vector<Iter_t> &index)
+{
+    typedef util::value_iter<Iter_t> value_t;
+
+    size_t pos_dest = 0;
+    size_t pos_src = 0;
+    size_t pos_in_vector = 0;
+    size_t nelem = index.size();
+    Iter_t it_dest, it_src;
+
+    while (pos_in_vector < nelem)
+    {
+        while (pos_in_vector < nelem and
+               (size_t(index[pos_in_vector] - global_first)) == pos_in_vector)
+        {
+            ++pos_in_vector;
+        };
+
+        if (pos_in_vector == nelem) return;
+        pos_dest = pos_src = pos_in_vector;
+        it_dest = global_first + pos_dest;
+        value_t Aux = std::move(*it_dest);
+
+        while ((pos_src = (size_t(index[pos_dest] - global_first)))
+               != pos_in_vector)
+        {
+            index[pos_dest] = it_dest;
+            it_src = global_first + pos_src;
+            *it_dest = std::move(*it_src);
+            it_dest = it_src;
+            pos_dest = pos_src;
+        };
+
+        *it_dest = std::move(Aux);
+        index[pos_dest] = it_dest;
+        ++pos_in_vector;
+    };
+};
+
+template<class func, class Iter_t, class Compare = compare_iter<Iter_t> >
+static void indirect_sort(func method, Iter_t first, Iter_t last, Compare comp)
+{
+    auto nelem = (last - first);
+    assert(nelem >= 0);
+    if (nelem < 2) return;
+    std::vector<Iter_t> index;
+    index.reserve((size_t) nelem);
+    create_index(first, last, index);
+    less_ptr_no_null<Iter_t, Compare> index_comp(comp);
+    method(index.begin(), index.end(), index_comp);
+    sort_index(first, index);
+};
+
+//
+//****************************************************************************
+};//    End namespace common
+};//    End namespace sort
+};//    End namespace boost
+//****************************************************************************
+//
+#endif
diff --git a/vendor/boost/sort/common/merge_four.hpp b/vendor/boost/sort/common/merge_four.hpp
new file mode 100644
index 00000000..edfb2ffc
--- /dev/null
+++ b/vendor/boost/sort/common/merge_four.hpp
@@ -0,0 +1,327 @@
+//----------------------------------------------------------------------------
+/// @file merge_four.hpp
+/// @brief This file have the functions for to merge 4 buffers
+///
+/// @author Copyright (c) 2016 Francisco José Tapia (fjtapia@gmail.com )\n
+///         Distributed under the Boost Software License, Version 1.0.\n
+///         ( See accompanying file LICENSE_1_0.txt or copy at
+///           http://www.boost.org/LICENSE_1_0.txt  )
+/// @version 0.1
+///
+/// @remarks
+//-----------------------------------------------------------------------------
+#ifndef __BOOST_SORT_PARALLEL_DETAIL_UTIL_MERGE_FOUR_HPP
+#define __BOOST_SORT_PARALLEL_DETAIL_UTIL_MERGE_FOUR_HPP
+
+#include <boost/sort/common/util/traits.hpp>
+#include <boost/sort/common/range.hpp>
+#include <functional>
+#include <iterator>
+#include <memory>
+#include <vector>
+
+namespace boost
+{
+namespace sort
+{
+namespace common
+{
+
+//
+//############################################################################
+//                                                                          ##
+//                       F U S I O N     O F                                ##
+//                                                                          ##
+//              F O U R     E L E M E N T S    R A N G E                    ##
+//                                                                          ##
+//############################################################################
+//
+
+//-----------------------------------------------------------------------------
+//  function : less_range
+/// @brief Compare the elements pointed by it1 and it2, and if they
+///        are equals, compare their position, doing a stable comparison
+///
+/// @param it1 : iterator to the first element
+/// @param pos1 : position of the object pointed by it1
+/// @param it2 : iterator to the second element
+/// @param pos2 : position of the element pointed by it2
+/// @param comp : comparison object
+/// @return result of the comparison
+//-----------------------------------------------------------------------------
+template<class Iter_t, class Compare = typename util::compare_iter<Iter_t> >
+inline bool less_range(Iter_t it1, uint32_t pos1, Iter_t it2, uint32_t pos2,
+                       Compare comp = Compare())
+{
+    return (comp(*it1, *it2)) ? true :
+           (pos2 < pos1) ? false : not (comp(*it2, *it1));
+};
+
+//-----------------------------------------------------------------------------
+//  function : full_merge4
+/// @brief Merge four ranges
+///
+/// @param dest: range where move the elements merged. Their size must be
+///              greater or equal than the sum of the sizes of the ranges
+///              in vrange_input
+/// @param vrange_input : array of ranges to merge
+/// @param nrange_input : number of ranges in vrange_input
+/// @param comp : comparison object
+/// @return range with all the elements moved with the size adjusted
+//-----------------------------------------------------------------------------
+template<class Iter1_t, class Iter2_t, class Compare>
+range<Iter1_t> full_merge4(const range<Iter1_t> &rdest,
+                           range<Iter2_t> vrange_input[4],
+                           uint32_t nrange_input, Compare comp)
+{
+    typedef range<Iter1_t> range1_t;
+    typedef util::value_iter<Iter1_t> type1;
+    typedef util::value_iter<Iter2_t> type2;
+    static_assert (std::is_same< type1, type2 >::value,
+                    "Incompatible iterators\n");
+
+    size_t ndest = 0;
+    uint32_t i = 0;
+    while (i < nrange_input)
+    {
+        if (vrange_input[i].size() != 0)
+        {
+            ndest += vrange_input[i++].size();
+        }
+        else
+        {
+            for (uint32_t k = i + 1; k < nrange_input; ++k)
+            {
+                vrange_input[k - 1] = vrange_input[k];
+            };
+            --nrange_input;
+        };
+    };
+
+    if (nrange_input == 0) return range1_t(rdest.first, rdest.first);
+    if (nrange_input == 1) return move_forward(rdest, vrange_input[0]);
+    if (nrange_input == 2)
+    {
+        return merge(rdest, vrange_input[0], vrange_input[1], comp);
+    };
+
+    //------------------------------------------------------------------------
+    // Initial sort
+    //------------------------------------------------------------------------
+    uint32_t pos[4] =
+    { 0, 1, 2, 3 }, npos = nrange_input;
+
+    //-----------------------------------------------------------------------
+    // thanks to Steven Ross by their suggestion about the optimal
+    // sorting networks
+    //-----------------------------------------------------------------------
+    if (less_range(vrange_input[pos[1]].first, pos[1],
+                    vrange_input[pos[0]].first, pos[0], comp))
+    {
+        std::swap(pos[0], pos[1]);
+    };
+    if (npos == 4 and less_range(vrange_input[pos[3]].first, pos[3],
+                                 vrange_input[pos[2]].first, pos[2], comp))
+    {
+        std::swap(pos[3], pos[2]);
+    };
+    if (less_range (vrange_input[pos[2]].first, pos[2],
+                    vrange_input[pos[0]].first, pos[0], comp))
+    {
+        std::swap(pos[0], pos[2]);
+    };
+    if (npos == 4
+                    and less_range (vrange_input[pos[3]].first, pos[3],
+                                    vrange_input[pos[1]].first, pos[1], comp))
+    {
+        std::swap(pos[1], pos[3]);
+    };
+    if (less_range (vrange_input[pos[2]].first, pos[2],
+                    vrange_input[pos[1]].first, pos[1], comp))
+    {
+        std::swap(pos[1], pos[2]);
+    };
+
+    Iter1_t it_dest = rdest.first;
+    while (npos > 2)
+    {
+        *(it_dest++) = std::move(*(vrange_input[pos[0]].first++));
+        if (vrange_input[pos[0]].size() == 0)
+        {
+            pos[0] = pos[1];
+            pos[1] = pos[2];
+            pos[2] = pos[3];
+            --npos;
+        }
+        else
+        {
+            if (less_range(vrange_input[pos[1]].first, pos[1],
+                            vrange_input[pos[0]].first, pos[0], comp))
+            {
+                std::swap(pos[0], pos[1]);
+                if (less_range(vrange_input[pos[2]].first, pos[2],
+                                vrange_input[pos[1]].first, pos[1], comp))
+                {
+                    std::swap(pos[1], pos[2]);
+                    if (npos == 4
+                                    and less_range(vrange_input[pos[3]].first,
+                                                    pos[3],
+                                                    vrange_input[pos[2]].first,
+                                                    pos[2], comp))
+                    {
+                        std::swap(pos[2], pos[3]);
+                    };
+                };
+            };
+        };
+    };
+
+    range1_t raux1(rdest.first, it_dest), raux2(it_dest, rdest.last);
+    if (pos[0] < pos[1])
+    {
+        return concat(raux1,merge(raux2, vrange_input[pos[0]], 
+                                  vrange_input[pos[1]], comp));
+    }
+    else
+    {
+        return concat(raux1, merge (raux2, vrange_input[pos[1]], 
+                                    vrange_input[pos[0]], comp));
+    };
+};
+
+//-----------------------------------------------------------------------------
+//  function : uninit_full_merge4
+/// @brief Merge four ranges and put the result in uninitialized memory
+///
+/// @param dest: range where create and move the elements merged. Their
+///              size must be greater or equal than the sum of the sizes
+///              of the ranges in the array R
+/// @param vrange_input : array of ranges to merge
+/// @param nrange_input : number of ranges in vrange_input
+/// @param comp : comparison object
+/// @return range with all the elements move with the size adjusted
+//-----------------------------------------------------------------------------
+template<class Value_t, class Iter_t, class Compare>
+range<Value_t *> uninit_full_merge4(const range<Value_t *> &dest,
+                                    range<Iter_t> vrange_input[4],
+                                    uint32_t nrange_input, Compare comp)
+{
+    typedef util::value_iter<Iter_t> type1;
+    static_assert (std::is_same< type1, Value_t >::value,
+                    "Incompatible iterators\n");
+
+    size_t ndest = 0;
+    uint32_t i = 0;
+    while (i < nrange_input)
+    {
+        if (vrange_input[i].size() != 0)
+        {
+            ndest += vrange_input[i++].size();
+        }
+        else
+        {
+            for (uint32_t k = i + 1; k < nrange_input; ++k)
+            {
+                vrange_input[k - 1] = vrange_input[k];
+            };
+            --nrange_input;
+        };
+    };
+    if (nrange_input == 0) return range<Value_t *>(dest.first, dest.first);
+    if (nrange_input == 1) return move_construct(dest, vrange_input[0]);
+    if (nrange_input == 2)
+    {
+        return merge_construct(dest, vrange_input[0], vrange_input[1], comp);
+    };
+
+    //------------------------------------------------------------------------
+    // Initial sort
+    //------------------------------------------------------------------------
+    uint32_t pos[4] = { 0, 1, 2, 3 }, npos = nrange_input;
+
+    //-----------------------------------------------------------------------
+    // thanks to Steven Ross by their suggestion about the optimal
+    // sorting networks
+    //-----------------------------------------------------------------------
+    if (less_range(vrange_input[pos[1]].first, pos[1],
+                    vrange_input[pos[0]].first, pos[0], comp))
+    {
+        std::swap(pos[0], pos[1]);
+    };
+    if (npos == 4  and less_range(vrange_input[pos[3]].first, pos[3],
+                                  vrange_input[pos[2]].first, pos[2], comp))
+    {
+        std::swap(pos[3], pos[2]);
+    };
+    if (less_range(vrange_input[pos[2]].first, pos[2],
+                    vrange_input[pos[0]].first, pos[0], comp))
+    {
+        std::swap(pos[0], pos[2]);
+    };
+    if (npos == 4 and less_range(vrange_input[pos[3]].first, pos[3],
+                                 vrange_input[pos[1]].first, pos[1], comp))
+    {
+        std::swap(pos[1], pos[3]);
+    };
+    if (less_range(vrange_input[pos[2]].first, pos[2],
+                    vrange_input[pos[1]].first, pos[1], comp))
+    {
+        std::swap(pos[1], pos[2]);
+    };
+
+    Value_t *it_dest = dest.first;
+    while (npos > 2)
+    {
+        util::construct_object(&(*(it_dest++)),
+                        std::move(*(vrange_input[pos[0]].first++)));
+        if (vrange_input[pos[0]].size() == 0)
+        {
+            pos[0] = pos[1];
+            pos[1] = pos[2];
+            pos[2] = pos[3];
+            --npos;
+        }
+        else
+        {
+            if (less_range (vrange_input[pos[1]].first, pos[1],
+                            vrange_input[pos[0]].first, pos[0], comp))
+            {
+                std::swap(pos[0], pos[1]);
+                if (less_range (vrange_input[pos[2]].first, pos[2],
+                                vrange_input[pos[1]].first, pos[1], comp))
+                {
+                    std::swap(pos[1], pos[2]);
+                    if (npos == 4 and less_range(vrange_input[pos[3]].first,
+                                                 pos[3],
+                                                 vrange_input[pos[2]].first,
+                                                 pos[2], comp))
+                    {
+                        std::swap(pos[2], pos[3]);
+                    };
+                };
+            };
+        };
+    }; // end while (npos > 2)
+
+    range<Value_t *> raux1(dest.first, it_dest), raux2(it_dest, dest.last);
+    if (pos[0] < pos[1])
+    {
+        return concat(raux1,
+                      merge_construct(raux2, vrange_input[pos[0]],
+                                      vrange_input[pos[1]], comp));
+    }
+    else
+    {
+        return concat(raux1,
+                      merge_construct(raux2, vrange_input[pos[1]],
+                                      vrange_input[pos[0]], comp));
+    };
+};
+
+//****************************************************************************
+};//    End namespace common
+};//    End namespace sort
+};//    End namespace boost
+//****************************************************************************
+//
+#endif
diff --git a/vendor/boost/sort/common/merge_vector.hpp b/vendor/boost/sort/common/merge_vector.hpp
new file mode 100644
index 00000000..84afea5a
--- /dev/null
+++ b/vendor/boost/sort/common/merge_vector.hpp
@@ -0,0 +1,196 @@
+//----------------------------------------------------------------------------
+/// @file merge_vector.hpp
+/// @brief In this file have the functions for to do a stable merge of
+//         ranges, in a vector
+///
+/// @author Copyright (c) 2016 Francisco Jose Tapia (fjtapia@gmail.com )\n
+///         Distributed under the Boost Software License, Version 1.0.\n
+///         ( See accompanying file LICENSE_1_0.txt or copy at
+///           http://www.boost.org/LICENSE_1_0.txt  )
+/// @version 0.1
+///
+/// @remarks
+//-----------------------------------------------------------------------------
+#ifndef __BOOST_SORT_PARALLEL_DETAIL_UTIL_MERGE_VECTOR_HPP
+#define __BOOST_SORT_PARALLEL_DETAIL_UTIL_MERGE_VECTOR_HPP
+
+#include <boost/sort/common/merge_four.hpp>
+#include <functional>
+#include <iterator>
+#include <memory>
+#include <type_traits>
+#include <vector>
+
+namespace boost
+{
+namespace sort
+{
+namespace common
+{
+
+//############################################################################
+//                                                                          ##
+//                       F U S I O N     O F                                ##
+//                                                                          ##
+//              A  V E C T O R   O F    R A N G E S                         ##
+//                                                                          ##
+//############################################################################
+
+//
+//-----------------------------------------------------------------------------
+//  function : merge_level4
+/// @brief merge the ranges in the vector v_input with the full_merge4 function.
+///        The v_output vector is used as auxiliary memory in the internal
+///        process. The final results is in the dest range.
+///        All the ranges of v_output are inside the range dest
+/// @param dest : range where move the elements merged
+/// @param v_input : vector of ranges to merge
+/// @param v_output : vector of ranges obtained
+/// @param comp : comparison object
+/// @return range with all the elements moved
+//-----------------------------------------------------------------------------
+template<class Iter1_t, class Iter2_t, class Compare>
+void merge_level4(range<Iter1_t> dest, std::vector<range<Iter2_t> > &v_input,
+                  std::vector<range<Iter1_t> > &v_output, Compare comp)
+{
+    typedef range<Iter1_t> range1_t;
+    typedef util::value_iter<Iter1_t> type1;
+    typedef util::value_iter<Iter2_t> type2;
+    static_assert (std::is_same< type1, type2 >::value,
+                    "Incompatible iterators\n");
+
+    v_output.clear();
+    if (v_input.size() == 0) return;
+    if (v_input.size() == 1)
+    {
+        v_output.emplace_back(move_forward(dest, v_input[0]));
+        return;
+    };
+
+    uint32_t nrange = v_input.size();
+    uint32_t pos_ini = 0;
+    while (pos_ini < v_input.size())
+    {
+        uint32_t nmerge = (nrange + 3) >> 2;
+        uint32_t nelem = (nrange + nmerge - 1) / nmerge;
+        range1_t rz = full_merge4(dest, &v_input[pos_ini], nelem, comp);
+        v_output.emplace_back(rz);
+        dest.first = rz.last;
+        pos_ini += nelem;
+        nrange -= nelem;
+    };
+    return;
+};
+//
+//-----------------------------------------------------------------------------
+//  function : uninit_merge_level4
+/// @brief merge the ranges moving the objects and constructing them in
+///        uninitialized memory, in the vector v_input
+///        using full_merge4. The v_output vector is used as auxiliary memory
+///        in the internal process. The final results is in the dest range.
+///        All the ranges of v_output are inside the range dest
+///
+/// @param dest : range where move the elements merged
+/// @param v_input : vector of ranges to merge
+/// @param v_output : vector of ranges obtained
+/// @param comp : comparison object
+/// @return range with all the elements moved and constructed
+//-----------------------------------------------------------------------------
+template<class Value_t, class Iter_t, class Compare>
+void uninit_merge_level4(range<Value_t *> dest,
+                         std::vector<range<Iter_t> > &v_input,
+                         std::vector<range<Value_t *> > &v_output, Compare comp)
+{
+    typedef range<Value_t *> range1_t;
+    typedef util::value_iter<Iter_t> type1;
+    static_assert (std::is_same< type1, Value_t >::value,
+                    "Incompatible iterators\n");
+
+    v_output.clear();
+    if (v_input.size() == 0) return;
+    if (v_input.size() == 1)
+    {
+        v_output.emplace_back(move_construct(dest, v_input[0]));
+        return;
+    };
+
+    uint32_t nrange = v_input.size();
+    uint32_t pos_ini = 0;
+    while (pos_ini < v_input.size())
+    {
+        uint32_t nmerge = (nrange + 3) >> 2;
+        uint32_t nelem = (nrange + nmerge - 1) / nmerge;
+        range1_t rz = uninit_full_merge4(dest, &v_input[pos_ini], nelem, comp);
+        v_output.emplace_back(rz);
+        dest.first = rz.last;
+        pos_ini += nelem;
+        nrange -= nelem;
+    };
+    return;
+};
+//
+//-----------------------------------------------------------------------------
+//  function : merge_vector4
+/// @brief merge the ranges in the vector v_input using the merge_level4
+///        function. The v_output vector is used as auxiliary memory in the
+///        internal process
+///        The final results is in the range_output range.
+///        All the ranges of v_output are inside the range range_output
+///        All the ranges of v_input are inside the range range_input
+/// @param range_input : range including all the ranges of v_input
+/// @param ange_output : range including all the elements of v_output
+/// @param v_input : vector of ranges to merge
+/// @param v_output : vector of ranges obtained
+/// @param comp : comparison object
+/// @return range with all the elements moved
+//-----------------------------------------------------------------------------
+template<class Iter1_t, class Iter2_t, class Compare>
+range<Iter2_t> merge_vector4(range<Iter1_t> range_input,
+                             range<Iter2_t> range_output,
+                             std::vector<range<Iter1_t> > &v_input,
+                             std::vector<range<Iter2_t> > &v_output,
+                             Compare comp)
+{
+    typedef range<Iter2_t> range2_t;
+    typedef util::value_iter<Iter1_t> type1;
+    typedef util::value_iter<Iter2_t> type2;
+    static_assert (std::is_same< type1, type2 >::value,
+                    "Incompatible iterators\n");
+
+    v_output.clear();
+    if (v_input.size() == 0)
+    {
+        return range2_t(range_output.first, range_output.first);
+    };
+    if (v_input.size() == 1)
+    {
+        return move_forward(range_output, v_input[0]);
+    };
+    bool sw = false;
+    uint32_t nrange = v_input.size();
+
+    while (nrange > 1)
+    {
+        if (sw)
+        {
+            merge_level4(range_input, v_output, v_input, comp);
+            sw = false;
+            nrange = v_input.size();
+        }
+        else
+        {
+            merge_level4(range_output, v_input, v_output, comp);
+            sw = true;
+            nrange = v_output.size();
+        };
+    };
+    return (sw) ? v_output[0] : move_forward(range_output, v_input[0]);
+};
+
+//****************************************************************************
+};//    End namespace common
+};//    End namespace sort
+};//    End namespace boost
+//****************************************************************************
+//
+#endif
diff --git a/vendor/boost/sort/common/range.hpp b/vendor/boost/sort/common/range.hpp
new file mode 100644
index 00000000..072d98a9
--- /dev/null
+++ b/vendor/boost/sort/common/range.hpp
@@ -0,0 +1,399 @@
+//----------------------------------------------------------------------------
+/// @file range.hpp
+/// @brief Define a range [first, last), and the associated operations
+///
+/// @author Copyright (c) 2016 Francisco José Tapia (fjtapia@gmail.com )\n
+///         Distributed under the Boost Software License, Version 1.0.\n
+///         ( See accompanyingfile LICENSE_1_0.txt or copy at
+///           http://www.boost.org/LICENSE_1_0.txt  )
+/// @version 0.1
+///
+/// @remarks
+//-----------------------------------------------------------------------------
+#ifndef __BOOST_SORT_PARALLEL_DETAIL_UTIL_RANGE_HPP
+#define __BOOST_SORT_PARALLEL_DETAIL_UTIL_RANGE_HPP
+
+#include <boost/sort/common/util/algorithm.hpp>
+#include <boost/sort/common/util/merge.hpp>
+#include <boost/sort/common/util/traits.hpp>
+#include <cassert>
+#include <functional>
+#include <memory>
+#include <vector>
+
+namespace boost
+{
+namespace sort
+{
+namespace common
+{
+
+///---------------------------------------------------------------------------
+/// @struct range
+/// @brief this represent a range between two iterators
+/// @remarks
+//----------------------------------------------------------------------------
+template <class Iter_t>
+struct range
+{
+    Iter_t first, last;
+    //
+    //------------------------------------------------------------------------
+    //  function : range
+    /// @brief  empty constructor
+    //------------------------------------------------------------------------
+    range(void) { };
+    //
+    //------------------------------------------------------------------------
+    //  function : range
+    /// @brief  constructor with two parameters
+    /// @param frs : iterator to the first element
+    /// @param lst : iterator to the last element
+    //-----------------------------------------------------------------------
+    range(const Iter_t &frs, const Iter_t &lst): first(frs), last(lst) { };
+    //
+    //-----------------------------------------------------------------------
+    //  function : empty
+    /// @brief indicate if the range is empty
+    /// @return  true : empty false : not empty
+    //-----------------------------------------------------------------------
+    bool empty(void) const { return (first == last); };
+    //
+    //-----------------------------------------------------------------------
+    //  function : not_empty
+    /// @brief indicate if the range is not empty
+    /// @return  true : not empty false : empty
+    //-----------------------------------------------------------------------
+    bool not_empty(void) const {return (first != last); };
+    //
+    //-----------------------------------------------------------------------
+    //  function : valid
+    /// @brief  Indicate if the range is well constructed, and valid
+    /// @return true : valid,  false : not valid
+    //-----------------------------------------------------------------------
+    bool valid(void) const { return ((last - first) >= 0); };
+    //
+    //-----------------------------------------------------------------------
+    //  function : size
+    /// @brief  return the size of the range
+    /// @return size
+    //-----------------------------------------------------------------------
+    size_t size(void) const { return (last - first); };
+    //
+    //------------------------------------------------------------------------
+    //  function : front
+    /// @brief return an iterator to the first element of the range
+    /// @return iterator
+    //-----------------------------------------------------------------------
+    Iter_t front(void) const { return first; };
+    //
+    //-------------------------------------------------------------------------
+    //  function : back
+    /// @brief return an iterator to the last element of the range
+    /// @return iterator
+    //-------------------------------------------------------------------------
+    Iter_t back(void) const {return (last - 1); };
+};
+
+//
+//-----------------------------------------------------------------------------
+//  function : concat
+/// @brief concatenate two contiguous ranges
+/// @param it1 : first range
+/// @param it2 : second range
+/// @return  range resulting of the concatenation
+//-----------------------------------------------------------------------------
+template<class Iter_t>
+inline range<Iter_t> concat(const range<Iter_t> &it1, const range<Iter_t> &it2)
+{
+    return range<Iter_t>(it1.first, it2.last);
+}
+;
+//
+//-----------------------------------------------------------------------------
+//  function : move_forward
+/// @brief Move initialized objets from the range src to dest
+/// @param dest : range where move the objects
+/// @param src : range from where move the objects
+/// @return range with the objects moved and the size adjusted
+//-----------------------------------------------------------------------------
+template <class Iter1_t, class Iter2_t>
+inline range<Iter2_t> move_forward(const range<Iter2_t> &dest,
+                                   const range<Iter1_t> &src)
+{
+    assert(dest.size() >= src.size());
+    Iter2_t it_aux = util::move_forward(dest.first, src.first, src.last);
+    return range<Iter2_t>(dest.first, it_aux);
+};
+//
+//-----------------------------------------------------------------------------
+//  function : move_backward
+/// @brief Move initialized objets from the range src to dest
+/// @param dest : range where move the objects
+/// @param src : range from where move the objects
+/// @return range with the objects moved and the size adjusted
+//-----------------------------------------------------------------------------
+template <class Iter1_t, class Iter2_t>
+inline range<Iter2_t> move_backward(const range<Iter2_t> &dest,
+                                    const range<Iter1_t> &src)
+{
+    assert(dest.size() >= src.size());
+    Iter2_t it_aux = util::move_backward(dest.first + src.size(), src.first,
+                    src.last);
+    return range<Iter2_t>(dest.first, dest.src.size());
+};
+
+//-----------------------------------------------------------------------------
+//  function : uninit_move
+/// @brief Move uninitialized objets from the range src creating them in  dest
+///
+/// @param dest : range where move and create the objects
+/// @param src : range from where move the objects
+/// @return range with the objects moved and the size adjusted
+//-----------------------------------------------------------------------------
+template<class Iter_t, class Value_t = util::value_iter<Iter_t> >
+inline range<Value_t*> move_construct(const range<Value_t*> &dest,
+                                      const range<Iter_t> &src)
+{
+    Value_t *ptr_aux = util::move_construct(dest.first, src.first, src.last);
+    return range<Value_t*>(dest.first, ptr_aux);
+};
+//
+//-----------------------------------------------------------------------------
+//  function : destroy
+/// @brief destroy a range of objects
+/// @param rng : range to destroy
+//-----------------------------------------------------------------------------
+template<class Iter_t>
+inline void destroy(range<Iter_t> rng)
+{
+    util::destroy(rng.first, rng.last);
+};
+//
+//-----------------------------------------------------------------------------
+//  function : initialize
+/// @brief initialize a range of objects with the object val moving across them
+/// @param rng : range of elements not initialized
+/// @param val : object used for the initialization
+/// @return range initialized
+//-----------------------------------------------------------------------------
+template<class Iter_t, class Value_t = util::value_iter<Iter_t> >
+inline range<Iter_t> initialize(const range<Iter_t> &rng, Value_t &val)
+{
+    util::initialize(rng.first, rng.last, val);
+    return rng;
+};
+//
+//-----------------------------------------------------------------------------
+//  function : is_mergeable
+/// @brief : indicate if two ranges have a possible merge
+/// @param src1 : first range
+/// @param src2 : second range
+/// @param comp : object for to compare elements
+/// @return true : they can be merged
+///         false : they can't be merged
+//-----------------------------------------------------------------------------
+template<class Iter1_t, class Iter2_t, class Compare>
+inline bool is_mergeable(const range<Iter1_t> &src1, const range<Iter2_t> &src2,
+                         Compare comp)
+{
+    //------------------------------------------------------------------------
+    //                  Metaprogramming
+    //------------------------------------------------------------------------
+    typedef util::value_iter<Iter1_t> type1;
+    typedef util::value_iter<Iter2_t> type2;
+    static_assert (std::is_same< type1, type2 >::value,
+                    "Incompatible iterators\n");
+    //------------------------------------------------------------------------
+    //                 Code
+    //------------------------------------------------------------------------
+    return comp(*(src2.front()), *(src1.back()));
+};
+//
+//-----------------------------------------------------------------------------
+//  function : is_mergeable_stable
+/// @brief : indicate if two ranges have a possible merge
+/// @param src1 : first range
+/// @param src2 : second range
+/// @param comp : object for to compare elements
+/// @return true : they can be merged
+///         false : they can't be merged
+//-----------------------------------------------------------------------------
+template<class Iter1_t, class Iter2_t, class Compare>
+inline bool is_mergeable_stable(const range<Iter1_t> &src1,
+                                const range<Iter2_t> &src2, Compare comp)
+{
+    //------------------------------------------------------------------------
+    //                  Metaprogramming
+    //------------------------------------------------------------------------
+    typedef util::value_iter<Iter1_t> type1;
+    typedef util::value_iter<Iter2_t> type2;
+    static_assert (std::is_same< type1, type2 >::value,
+                    "Incompatible iterators\n");
+    //------------------------------------------------------------------------
+    //                 Code
+    //------------------------------------------------------------------------
+    return not comp(*(src1.back()), *(src2.front()));
+};
+//
+//-----------------------------------------------------------------------------
+//  function : merge
+/// @brief Merge two contiguous ranges src1 and src2, and put the result in
+///        the range dest, returning the range merged
+///
+/// @param dest : range where locate the lements merged. the size of dest
+///               must be  greater or equal than the sum of the sizes of
+///               src1 and src2
+/// @param src1 : first range to merge
+/// @param src2 : second range to merge
+/// @param comp : comparison object
+/// @return range with the elements merged and the size adjusted
+//-----------------------------------------------------------------------------
+template<class Iter1_t, class Iter2_t, class Iter3_t, class Compare>
+inline range<Iter3_t> merge(const range<Iter3_t> &dest,
+                            const range<Iter1_t> &src1,
+                            const range<Iter2_t> &src2, Compare comp)
+{
+    Iter3_t it_aux = util::merge(src1.first, src1.last, src2.first, src2.last,
+                    dest.first, comp);
+    return range<Iter3_t>(dest.first, it_aux);
+};
+
+//-----------------------------------------------------------------------------
+//  function : merge_construct
+/// @brief Merge two contiguous uninitialized ranges src1 and src2, and create
+///        and move the result in the uninitialized range dest, returning the
+///        range merged
+//
+/// @param dest : range where locate the elements merged. the size of dest
+///               must be  greater or equal than the sum of the sizes of
+///               src1 and src2. Initially is uninitialize memory
+/// @param src1 : first range to merge
+/// @param src2 : second range to merge
+/// @param comp : comparison object
+/// @return range with the elements merged and the size adjusted
+//-----------------------------------------------------------------------------
+template<class Iter1_t, class Iter2_t, class Value_t, class Compare>
+inline range<Value_t *> merge_construct(const range<Value_t *> &dest,
+                                        const range<Iter1_t> &src1,
+                                        const range<Iter2_t> &src2,
+                                        Compare comp)
+{
+    Value_t * ptr_aux = util::merge_construct(src1.first, src1.last, src2.first,
+                    src2.last, dest.first, comp);
+    return range<Value_t*>(dest.first, ptr_aux);
+};
+//
+//---------------------------------------------------------------------------
+//  function : half_merge
+/// @brief : Merge two initialized buffers. The first buffer is in a separate
+///          memory
+//
+/// @param dest : range where finish the two buffers merged
+/// @param src1 : first range to merge in a separate memory
+/// @param src2 : second range to merge, in the final part of the
+///               range where deposit the final results
+/// @param comp : object for compare two elements of the type pointed
+///               by the Iter1_t and Iter2_t
+/// @return : range with the two buffers merged
+//---------------------------------------------------------------------------
+template<class Iter1_t, class Iter2_t, class Compare>
+inline range<Iter2_t> merge_half(const range<Iter2_t> &dest,
+                                 const range<Iter1_t> &src1,
+                                 const range<Iter2_t> &src2, Compare comp)
+{
+    Iter2_t it_aux = util::merge_half(src1.first, src1.last, src2.first,
+                    src2.last, dest.first, comp);
+    return range<Iter2_t>(dest.first, it_aux);
+};
+//
+//-----------------------------------------------------------------------------
+//  function : merge_uncontiguous
+/// @brief : merge two non contiguous ranges src1, src2, using the range
+///          aux as auxiliary memory. The results are in the original ranges
+//
+/// @param src1 : first range to merge
+/// @param src2 : second range to merge
+/// @param aux : auxiliary range used in the merge
+/// @param comp : object for to compare elements
+/// @return true : not changes done, false : changes in the buffers
+//-----------------------------------------------------------------------------
+template<class Iter1_t, class Iter2_t, class Iter3_t, class Compare>
+inline bool merge_uncontiguous(const range<Iter1_t> &src1,
+                               const range<Iter2_t> &src2,
+                               const range<Iter3_t> &aux, Compare comp)
+{
+    return util::merge_uncontiguous(src1.first, src1.last, src2.first,
+                    src2.last, aux.first, comp);
+};
+//
+//-----------------------------------------------------------------------------
+//  function : merge_contiguous
+/// @brief : merge two contiguous ranges ( src1, src2) using buf as
+///          auxiliary memory. The results are in the same ranges
+/// @param src1 : first range to merge
+/// @param src1 : second range to merge
+/// @param buf : auxiliary memory used in the merge
+/// @param comp : object for to compare elements
+/// @return true : not changes done,   false : changes in the buffers
+//-----------------------------------------------------------------------------
+template<class Iter1_t, class Iter2_t, class Compare>
+inline range<Iter1_t> merge_contiguous(const range<Iter1_t> &src1,
+                                       const range<Iter1_t> &src2,
+                                       const range<Iter2_t> &buf, Compare comp)
+{
+    util::merge_contiguous(src1.first, src1.last, src2.last, buf.first, comp);
+    return concat(src1, src2);
+};
+//
+//-----------------------------------------------------------------------------
+//  function : merge_flow
+/// @brief : merge two ranges, as part of a merge the ranges in a list. This
+///         function reduce the number of movements compared with inplace_merge
+///         when you need to merge a sequence of ranges.
+///         This function merge the ranges rbuf and rng2, and the results
+///          are in rng1 and rbuf
+//
+/// @param rng1 : range where locate the first elements of the merge
+/// @param rbuf : range which provide the first elements, and where store
+///               the last results of the merge
+/// @param rng2 : range which provide the last elements to merge
+/// @param comp : object for to compare elements
+/// @return true : not changes done,  false : changes in the buffers
+//-----------------------------------------------------------------------------
+template<class Iter1_t, class Iter2_t, class Compare>
+static void merge_flow(range<Iter1_t> rng1, range<Iter2_t> rbuf,
+                       range<Iter1_t> rng2, Compare cmp)
+{
+    //-------------------------------------------------------------------------
+    //                       Metaprogramming
+    //-------------------------------------------------------------------------
+    typedef util::value_iter<Iter1_t> type1;
+    typedef util::value_iter<Iter2_t> type2;
+    static_assert (std::is_same< type1, type2 >::value,
+                    "Incompatible iterators\n");
+
+    //-------------------------------------------------------------------------
+    //                       Code
+    //-------------------------------------------------------------------------
+    range<Iter2_t> rbx(rbuf);
+    range<Iter1_t> rx1(rng1), rx2(rng2);
+    assert(rbx.size() == rx1.size() and rx1.size() == rx2.size());
+    while (rx1.first != rx1.last)
+    {
+        *(rx1.first++) = (cmp(*rbx.first, *rx2.first)) ?
+                                                    std::move(*(rbx.first++)):
+                                                    std::move(*(rx2.first++));
+    };
+    if (rx2.first == rx2.last) return;
+    if (rbx.first == rbx.last) move_forward(rbuf, rng2);
+    else                       merge_half(rbuf, rx2, rbx, cmp);
+};
+
+//****************************************************************************
+};//    End namespace common
+};//    End namespace sort
+};//    End namespace boost
+//****************************************************************************
+//
+#endif
diff --git a/vendor/boost/sort/common/util/algorithm.hpp b/vendor/boost/sort/common/util/algorithm.hpp
new file mode 100644
index 00000000..db7607aa
--- /dev/null
+++ b/vendor/boost/sort/common/util/algorithm.hpp
@@ -0,0 +1,309 @@
+//----------------------------------------------------------------------------
+/// @file algorithm.hpp
+/// @brief low level functions of create, destroy, move and merge functions
+///
+/// @author Copyright (c) 2017 Francisco Jose Tapia (fjtapia@gmail.com )\n
+///         Distributed under the Boost Software License, Version 1.0.\n
+///         ( See accompanying file LICENSE_1_0.txt or copy at
+///           http://www.boost.org/LICENSE_1_0.txt  )
+/// @version 0.1
+///
+/// @remarks
+//-----------------------------------------------------------------------------
+#ifndef __BOOST_SORT_COMMON_UTIL_ALGORITHM_HPP
+#define __BOOST_SORT_COMMON_UTIL_ALGORITHM_HPP
+
+#include <algorithm>
+#include <functional>
+#include <iterator>
+#include <memory>
+#include <type_traits>
+#include <vector>
+#include <boost/sort/common/util/traits.hpp>
+
+namespace boost
+{
+namespace sort
+{
+namespace common
+{
+namespace util
+{
+//
+//###########################################################################
+//
+//                       I M P O R T A N T
+//
+// The functions of this file are for internal use only
+// All the operations are done with move operations, because the copy
+// operations are unnecesary
+//
+//###########################################################################
+//
+//----------------------------------------------------------------------------
+//
+//         F U N C T I O N S   I N   T H E   F I L E
+//
+//----------------------------------------------------------------------------
+//
+// static inline uint32_t nbits32 (uint32_t num) noexcept
+//
+// static inline uint32_t nbits64 (uint64_t num)
+//
+// template < class Value_t, class... Args >
+// inline void construct_object (Value_t *ptr, Args &&... args)
+//
+// template < class Value_t >
+// inline void destroy_object (Value_t *ptr)
+//
+// template < class Iter_t, class Value_t = value_iter<Iter_t> >
+// void initialize (Iter_t first, Iter_t last, Value_t && val)
+//
+// template < class Iter1_t, class Iter2_t >
+// Iter2_t move_forward (Iter2_t it_dest, Iter1_t first, Iter1_t last)
+//
+// template < class Iter1_t, class Iter2_t >
+// Iter2_t move_backward (Iter2_t it_dest, Iter1_t first, Iter1_t last)
+//
+// template < class Iter_t, class Value_t = value_iter< Iter_t > >
+// Value_t * move_construct (Value_t *ptr, Iter_t first, Iter_t last)
+//
+// template < class Iter_t >
+// void destroy (Iter_t first, const Iter_t last)
+//
+// template < class Iter_t >
+// void reverse (Iter_t first, const Iter_t last)
+//
+//----------------------------------------------------------------------------
+//
+//--------------------------------------------------------------------------
+//
+//                    G L O B A L     V A R I B L E S
+//
+//--------------------------------------------------------------------------
+//
+// this array represent the number of bits needed for to represent the
+// first 256 numbers
+static constexpr const uint32_t tmsb[256] =
+{ 0, 1, 2, 2, 3, 3, 3, 3, 4, 4, 4, 4, 4, 4, 4, 4, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5,
+                5, 5, 5, 5, 5, 5, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6,
+                6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 7, 7, 7, 7,
+                7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7,
+                7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7,
+                7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 8, 8, 8,
+                8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8,
+                8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8,
+                8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8,
+                8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8,
+                8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8,
+                8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8 };
+//
+//---------------------------------------------------------------------------
+//
+//                           F U N C T I O N S
+//
+//---------------------------------------------------------------------------
+//
+//---------------------------------------------------------------------------
+//  function : nbits32
+/// @brief Obtain the number of bits of a number equal or greater than num
+/// @param num : Number to examine
+/// @return Number of bits
+//---------------------------------------------------------------------------
+static inline uint32_t nbits32 (uint32_t num) noexcept
+{
+    int Pos = (num & 0xffff0000U) ? 16 : 0;
+    if ((num >> Pos) & 0xff00U) Pos += 8;
+    return (tmsb[num >> Pos] + Pos);
+}
+//
+//---------------------------------------------------------------------------
+//  function : nbits64
+/// @brief Obtain the number of bits of a number equal or greater than num
+/// @param num : Number to examine
+/// @exception none
+/// @return Number of bits
+//---------------------------------------------------------------------------
+static inline uint32_t nbits64(uint64_t num)noexcept
+{
+    uint32_t Pos = (num & 0xffffffff00000000ULL) ? 32 : 0;
+    if ((num >> Pos) & 0xffff0000ULL) Pos += 16;
+    if ((num >> Pos) & 0xff00ULL) Pos += 8;
+    return (tmsb[num >> Pos] + Pos);
+}
+//
+//-----------------------------------------------------------------------------
+//  function : construct_object
+/// @brief create an object in the memory specified by ptr
+///
+/// @param ptr : pointer to the memory where to create the object
+/// @param args : arguments to the constructor
+//-----------------------------------------------------------------------------
+template <class Value_t, class ... Args>
+inline void construct_object (Value_t *ptr, Args &&... args)
+{
+    (::new (static_cast<void *>(ptr)) Value_t(std::forward< Args > (args)...));
+};
+//
+//-----------------------------------------------------------------------------
+//  function : destroy_object
+/// @brief destroy an object in the memory specified by ptr
+/// @param ptr : pointer to the object to destroy
+//-----------------------------------------------------------------------------
+template<class Value_t>
+inline void destroy_object(Value_t *ptr)
+{
+    ptr->~Value_t();
+};
+//
+//-----------------------------------------------------------------------------
+//  function : initialize
+/// @brief initialize a range of objects with the object val moving across them
+///
+/// @param first : itertor to the first element to initialize
+/// @param last : iterator to the last element to initialize
+/// @param val : object used for the initialization
+//-----------------------------------------------------------------------------
+template <class Iter_t, class Value_t = value_iter<Iter_t> >
+inline void initialize (Iter_t first, Iter_t last, Value_t & val)
+{
+    //------------------------------------------------------------------------
+    //                  Metaprogramming
+    //------------------------------------------------------------------------
+    typedef value_iter<Iter_t> value_t;
+    static_assert (std::is_same< Value_t, value_t >::value,
+                    "Incompatible iterators\n");
+
+    //------------------------------------------------------------------------
+    //                 Code
+    //------------------------------------------------------------------------
+    if (first == last) return;
+    construct_object(&(*first), std::move(val));
+
+    Iter_t it1 = first, it2 = first + 1;
+    while (it2 != last)
+    {
+        construct_object(&(*(it2++)), std::move(*(it1++)));
+    };
+    val = std::move(*(last - 1));
+};
+//
+//-----------------------------------------------------------------------------
+//  function : move_forward
+/// @brief Move initialized objets
+/// @param it_dest : iterator to the final place of the objects
+/// @param first : iterator to the first element to move
+/// @param last : iterator to the last element to move
+/// @return Output iterator to the element past the last element
+///         moved (it_dest + (last - first))
+//-----------------------------------------------------------------------------
+template <class Iter1_t, class Iter2_t>
+inline Iter2_t move_forward (Iter2_t it_dest, Iter1_t first, Iter1_t last)
+{
+    //------------------------------------------------------------------------
+    //                  Metaprogramming
+    //------------------------------------------------------------------------
+    typedef value_iter<Iter1_t> value1_t;
+    typedef value_iter<Iter2_t> value2_t;
+    static_assert (std::is_same< value1_t, value2_t >::value,
+                    "Incompatible iterators\n");
+
+    //------------------------------------------------------------------------
+    //                 Code
+    //------------------------------------------------------------------------
+    while (first != last)
+    {   *it_dest++ = std::move(*first++);
+    }
+    return it_dest;
+
+};
+//
+//-----------------------------------------------------------------------------
+//  function : move_backard
+/// @brief Move initialized objets in reverse order
+/// @param it_dest : last iterator to the final place of the objects
+/// @param first : iterator to the first element to move
+/// @param last : iterator to the last element to move
+//-----------------------------------------------------------------------------
+template<class Iter1_t, class Iter2_t>
+inline Iter2_t move_backward(Iter2_t it_dest, Iter1_t  first, Iter1_t last)
+{
+    //------------------------------------------------------------------------
+    //                  Metaprogramming
+    //------------------------------------------------------------------------
+    typedef value_iter<Iter1_t> value1_t;
+    typedef value_iter<Iter2_t> value2_t;
+    static_assert (std::is_same< value1_t, value2_t >::value,
+                    "Incompatible iterators\n");
+
+    //------------------------------------------------------------------------
+    //                 Code
+    //------------------------------------------------------------------------
+    while (first != last)
+    {   *(--it_dest) = std::move (*(--last));
+    }
+    return it_dest;
+};
+
+//
+//-----------------------------------------------------------------------------
+//  function : move_construct
+/// @brief Move objets to uninitialized memory
+///
+/// @param ptr : pointer to the memory where to create the objects
+/// @param first : iterator to the first element to move
+/// @param last : iterator to the last element to move
+//-----------------------------------------------------------------------------
+template<class Iter_t, class Value_t = value_iter<Iter_t> >
+inline Value_t * move_construct(Value_t *ptr, Iter_t first, Iter_t last)
+{
+    //------------------------------------------------------------------------
+    //                  Metaprogramming
+    //------------------------------------------------------------------------
+    typedef typename iterator_traits<Iter_t>::value_type value2_t;
+    static_assert (std::is_same< Value_t, value2_t >::value,
+                    "Incompatible iterators\n");
+
+    //------------------------------------------------------------------------
+    //                    Code
+    //------------------------------------------------------------------------
+    while (first != last)
+    {
+        ::new (static_cast<void *>(ptr++)) Value_t(std::move(*(first++)));
+    };
+    return ptr;
+};
+//
+//-----------------------------------------------------------------------------
+//  function : destroy
+/// @brief destroy the elements between first and last
+/// @param first : iterator to the first element to destroy
+/// @param last : iterator to the last element to destroy
+//-----------------------------------------------------------------------------
+template<class Iter_t>
+inline void destroy(Iter_t first, const Iter_t last)
+{
+    while (first != last)
+        destroy_object(&(*(first++)));
+};
+//
+//-----------------------------------------------------------------------------
+//  function : reverse
+/// @brief destroy the elements between first and last
+/// @param first : iterator to the first element to destroy
+/// @param last : iterator to the last element to destroy
+//-----------------------------------------------------------------------------
+template<class Iter_t>
+inline void reverse(Iter_t first, Iter_t last)
+{
+    std::reverse ( first, last);
+};
+//
+//****************************************************************************
+};//    End namespace util
+};//    End namespace common
+};//    End namespace sort
+};//    End namespace boost
+//****************************************************************************
+//
+#endif
diff --git a/vendor/boost/sort/common/util/atomic.hpp b/vendor/boost/sort/common/util/atomic.hpp
new file mode 100644
index 00000000..15906fe5
--- /dev/null
+++ b/vendor/boost/sort/common/util/atomic.hpp
@@ -0,0 +1,98 @@
+//----------------------------------------------------------------------------
+/// @file atomic.hpp
+/// @brief Basic layer for to simplify the use of atomic functions
+/// @author Copyright(c) 2016 Francisco José Tapia (fjtapia@gmail.com )\n
+///         Distributed under the Boost Software License, Version 1.0.\n
+///         ( See accompanying file LICENSE_1_0.txt or copy at
+///           http://www.boost.org/LICENSE_1_0.txt  )
+/// @version 0.1
+///
+/// @remarks
+//-----------------------------------------------------------------------------
+#ifndef __BOOST_SORT_PARALLEL_DETAIL_UTIL_ATOMIC_HPP
+#define __BOOST_SORT_PARALLEL_DETAIL_UTIL_ATOMIC_HPP
+
+#include <atomic>
+#include <cassert>
+#include <type_traits>
+
+namespace boost
+{
+namespace sort
+{
+namespace common
+{
+namespace util
+{
+//-----------------------------------------------------------------------------
+//  function : atomic_read
+/// @brief make the atomic read of an atomic variable, using a memory model
+/// @param at_var : atomic variable to read
+/// @return value obtained
+//-----------------------------------------------------------------------------
+template<typename T>
+inline T atomic_read(std::atomic<T> &at_var)
+{
+    return std::atomic_load_explicit < T > (&at_var, std::memory_order_acquire);
+};
+//
+//-----------------------------------------------------------------------------
+//  function : atomic_add
+/// @brief Add a number to an atomic variable, using a memory model
+/// @param at_var : variable to add
+/// @param num : value to add to at_var
+/// @return result of the operation
+//-----------------------------------------------------------------------------
+template<typename T, typename T2>
+inline T atomic_add(std::atomic<T> &at_var, T2 num)
+{
+    static_assert (std::is_integral< T2 >::value, "Bad parameter");
+    return std::atomic_fetch_add_explicit <T> 
+                               (&at_var, (T) num, std::memory_order_acq_rel);
+};
+//
+//-----------------------------------------------------------------------------
+//  function : atomic_sub
+/// @brief Atomic subtract of an atomic variable using memory model
+/// @param at_var : Varibale to subtract
+/// @param num : value to sub to at_var
+/// @return result of the operation
+//-----------------------------------------------------------------------------
+template<typename T, typename T2>
+inline T atomic_sub(std::atomic<T> &at_var, T2 num)
+{
+    static_assert (std::is_integral< T2 >::value, "Bad parameter");
+    return std::atomic_fetch_sub_explicit <T> 
+                                (&at_var, (T) num, std::memory_order_acq_rel);
+};
+//
+//-----------------------------------------------------------------------------
+//  function : atomic_write
+/// @brief Write a value in an atomic variable using memory model
+/// @param at_var : varible to write
+/// @param num : value to write in at_var
+//-----------------------------------------------------------------------------
+template<typename T, typename T2>
+inline void atomic_write(std::atomic<T> &at_var, T2 num)
+{
+    static_assert (std::is_integral< T2 >::value, "Bad parameter");
+    std::atomic_store_explicit <T> 
+                                (&at_var, (T) num, std::memory_order_release);
+};
+template<typename T>
+struct counter_guard
+{
+    typedef std::atomic<T> atomic_t;
+    atomic_t &count;
+
+    counter_guard(atomic_t & counter): count(counter) { };
+    ~counter_guard() {atomic_sub(count, 1); };
+};
+//
+//****************************************************************************
+};// End namespace util
+};// End namespace common
+};// End namespace sort
+};// End namespace boost
+//****************************************************************************
+#endif
diff --git a/vendor/boost/sort/common/util/circular_buffer.hpp b/vendor/boost/sort/common/util/circular_buffer.hpp
new file mode 100644
index 00000000..2fc7e973
--- /dev/null
+++ b/vendor/boost/sort/common/util/circular_buffer.hpp
@@ -0,0 +1,572 @@
+//----------------------------------------------------------------------------
+/// @file   circular_buffer.hpp
+/// @brief  This file contains the implementation of the circular buffer
+///
+/// @author Copyright (c) 2010 2015 Francisco José Tapia (fjtapia@gmail.com )\n
+///         Distributed under the Boost Software License, Version 1.0.\n
+///         ( See accompanyingfile LICENSE_1_0.txt or copy at
+///           http://www.boost.org/LICENSE_1_0.txt  )
+/// @version 0.1
+///
+/// @remarks
+//-----------------------------------------------------------------------------
+#ifndef __BOOST_SORT_COMMON_UTIL_CIRCULAR_BUFFER_HPP
+#define __BOOST_SORT_COMMON_UTIL_CIRCULAR_BUFFER_HPP
+
+#include <memory>
+#include <cassert>
+#include <exception>
+#include <boost/sort/common/util/algorithm.hpp>
+#include <boost/sort/common/util/traits.hpp>
+
+namespace boost
+{
+namespace sort
+{
+namespace common
+{
+namespace util
+{
+
+//---------------------------------------------------------------------------
+/// @class  circular_buffer
+/// @brief  This class implement a circular buffer
+/// @remarks
+//---------------------------------------------------------------------------
+template <class Value_t, uint32_t Power2 = 11>
+struct circular_buffer
+{
+    //------------------------------------------------------------------------
+    //                          STATIC CHECK
+    //------------------------------------------------------------------------
+    static_assert ( Power2 != 0, "Wrong Power2");
+
+    //------------------------------------------------------------------------
+    //                          DEFINITIONS
+    //------------------------------------------------------------------------
+    typedef Value_t value_t;
+
+    //------------------------------------------------------------------------
+    //                          VARIABLES
+    //------------------------------------------------------------------------
+    const size_t NMAX = (size_t) 1 << Power2;
+    const size_t MASK = (NMAX - 1);
+    const size_t BLOCK_SIZE = NMAX >> 1;
+    const size_t LOG_BLOCK = Power2 - 1;
+    Value_t * ptr = nullptr;
+
+    //------------------------------------------------------------------------
+    // first and last are  the position of the first and last elements
+    // always are in the range [0, NMAX - 1]
+    //------------------------------------------------------------------------
+    size_t nelem, first_pos;
+    bool initialized;
+
+    //
+    //------------------------------------------------------------------------
+    //  function : circular_buffer
+    /// @brief  constructor of the class
+    //-----------------------------------------------------------------------
+    circular_buffer(void)
+    : ptr(nullptr), nelem(0), first_pos(0), initialized(false)
+    {
+        ptr = std::get_temporary_buffer < Value_t > (NMAX).first;
+        if (ptr == nullptr) throw std::bad_alloc();
+    };
+    //
+    //------------------------------------------------------------------------
+    //  function : ~circular_buffer
+    /// @brief destructor of the class
+    //-----------------------------------------------------------------------
+    ~circular_buffer()
+    {
+        if (initialized)
+        {   for (size_t i = 0; i < NMAX; ++i) (ptr + i)->~Value_t();
+            initialized = false;
+        };
+        std::return_temporary_buffer(ptr);
+    }
+    ;
+    //
+    //------------------------------------------------------------------------
+    //  function : initialize
+    /// @brief : initialize the memory of the buffer from the uninitialize
+    //           memory obtained from the temporary buffer
+    /// @param val : value used to initialize the memory
+    //-----------------------------------------------------------------------
+    void initialize(Value_t & val)
+    {
+        assert (initialized == false);
+        ::new (static_cast<void*>(ptr)) Value_t(std::move(val));
+        for (size_t i = 1; i < NMAX; ++i)
+            ::new (static_cast<void*>(ptr + i)) Value_t(std::move(ptr[i - 1]));
+        val = std::move(ptr[NMAX - 1]);
+        initialized = true;
+    };
+    //
+    //------------------------------------------------------------------------
+    //  function : destroy_all
+    /// @brief : destroy all the objects in the internal memory
+    //-----------------------------------------------------------------------
+    void destroy_all(void) { destroy(ptr, ptr + NMAX); };
+    //
+    //------------------------------------------------------------------------
+    //  function : get_buffer
+    /// @brief return the internal memory of the circular buffer
+    /// @return pointer to the internal memory of the buffer
+    //-----------------------------------------------------------------------
+    Value_t * get_buffer(void) { return ptr; };
+    //
+    //------------------------------------------------------------------------
+    //  function : empty
+    /// @brief return if the buffer is empty
+    /// @return true : empty
+    //-----------------------------------------------------------------------
+    bool empty(void) const {return (nelem == 0); };
+    //
+    //------------------------------------------------------------------------
+    //  function : full
+    /// @brief return if the buffer is full
+    /// @return true : full
+    //-----------------------------------------------------------------------
+    bool full(void) const { return (nelem == NMAX); };
+    //
+    //------------------------------------------------------------------------
+    //  function : size
+    /// @brief return the number of elements stored in the buffer
+    /// @return number of elements stored
+    //-----------------------------------------------------------------------
+    size_t size(void) const { return nelem;};
+    //
+    //------------------------------------------------------------------------
+    //  function : capacity
+    /// @brief : return the maximun capacity of the buffer
+    /// @return number of elements
+    //-----------------------------------------------------------------------
+    size_t capacity(void) const { return NMAX;};
+    //
+    //------------------------------------------------------------------------
+    //  function : free_size
+    /// @brief return the free positions in the buffer
+    /// @return number of elements
+    //-----------------------------------------------------------------------
+    size_t free_size(void) const  { return (NMAX - nelem); };
+    //
+    //------------------------------------------------------------------------
+    //  function : clear
+    /// @brief clear the buffer
+    //-----------------------------------------------------------------------
+    void clear(void)  { nelem = first_pos = 0; };
+    //
+    //------------------------------------------------------------------------
+    //  function : front
+    /// @brief return the first element of the buffer
+    /// @return reference to the first value
+    //-----------------------------------------------------------------------
+    Value_t & front(void)
+    {
+#ifdef __BS_DEBUG
+        assert (nelem > 0);
+#endif
+        return (ptr[first_pos]);
+    };
+    //
+    //------------------------------------------------------------------------
+    //  function :front
+    /// @brief return the first element of the buffer
+    /// @return const reference to the first value
+    //-----------------------------------------------------------------------
+    const Value_t & front(void) const
+    {
+#ifdef __BS_DEBUG
+        assert ( nelem > 0 );
+#endif
+        return (ptr[first_pos]);
+    };
+    //
+    //------------------------------------------------------------------------
+    //  function : back
+    /// @brief reference to the last value of the buffer
+    /// @return reference to the last value
+    //-----------------------------------------------------------------------
+    Value_t & back(void)
+    {
+#ifdef __BS_DEBUG
+        assert ( nelem > 0 );
+#endif
+        return (ptr[(first_pos + nelem - 1) & MASK]);
+    };
+    //
+    //------------------------------------------------------------------------
+    //  function : back
+    /// @brief reference to the last value of the buffer
+    /// @return const reference to the last value
+    //-----------------------------------------------------------------------
+    const Value_t & back(void) const
+    {
+#ifdef __BS_DEBUG
+        assert ( nelem > 0 );
+#endif
+        return (ptr[(first_pos + nelem - 1) & MASK]);
+    };
+    //
+    //------------------------------------------------------------------------
+    //  function : operator []
+    /// @brief positional access to the elements
+    /// @param pos rquested
+    /// @return reference to the element
+    //-----------------------------------------------------------------------
+    Value_t & operator[](uint32_t pos)
+    {
+#ifdef __BS_DEBUG
+        assert ( nelem > 0 );
+#endif
+        return ptr[(first_pos + pos) & MASK];
+    };
+    //
+    //------------------------------------------------------------------------
+    //  function : operator []
+    /// @brief positional access to the elements
+    /// @param pos rquested
+    /// @return const reference to the element
+    //-----------------------------------------------------------------------
+    const Value_t & operator[](uint32_t pos) const
+    {
+
+#ifdef __BS_DEBUG
+        assert ( nelem > 0 );
+#endif
+        return ptr[(first_pos + pos) & MASK];
+    };
+    //
+    //------------------------------------------------------------------------
+    //  function : push_front
+    /// @brief insert an element in the first position of the buffer
+    /// @param val : const value to insert
+    //-----------------------------------------------------------------------
+    void push_front(const Value_t & val)
+    {
+#ifdef __BS_DEBUG
+        assert ( nelem != NMAX);
+#endif
+        ++nelem;
+        first_pos = ((first_pos + MASK) & MASK);
+        ptr[first_pos] = val;
+
+    };
+    //
+    //------------------------------------------------------------------------
+    //  function : push_front
+    /// @brief insert an element in the first position of the buffer
+    /// @param val : rvalue to insert
+    //-----------------------------------------------------------------------
+    void push_front(Value_t && val)
+    {
+#ifdef __BS_DEBUG
+        assert ( nelem != NMAX);
+#endif
+        ++nelem;
+        first_pos = ((first_pos + MASK) & MASK);
+        ptr[first_pos] = val;
+    };
+    //
+    //------------------------------------------------------------------------
+    //  function : push_back
+    /// @brief insert an element in the last position of the buffer
+    /// @param val : value to insert
+    //-----------------------------------------------------------------------
+    void push_back(const Value_t & val)
+    {
+#ifdef __BS_DEBUG
+        assert ( nelem != NMAX);
+#endif
+        ptr[(first_pos + (nelem++)) & MASK] = val;
+    };
+    //
+    //------------------------------------------------------------------------
+    //  function : push_back
+    /// @brief insert an element in the last position of the buffer
+    /// @param val : value to insert
+    //-----------------------------------------------------------------------
+    void push_back(Value_t && val)
+    {
+#ifdef __BS_DEBUG
+        assert ( nelem != NMAX);
+#endif
+        ptr[(first_pos + (nelem++)) & MASK] = std::move(val);
+    };
+    //
+    //------------------------------------------------------------------------
+    //  function : pop_front
+    /// @brief remove the first element of the buffer
+    //-----------------------------------------------------------------------
+    void pop_front(void)
+    {
+#ifdef __BS_DEBUG
+        assert ( nelem > 0 );
+#endif
+        --nelem;
+        (++first_pos) &= MASK;
+    };
+    //
+    //------------------------------------------------------------------------
+    //  function : pop_back
+    /// @brief remove the last element of the buffer
+    //-----------------------------------------------------------------------
+    void pop_back(void)
+    {
+#ifdef __BS_DEBUG
+        assert ( nelem > 0 );
+#endif
+        --nelem;
+    };
+
+    template<class iter_t>
+    void pop_copy_front(iter_t it_dest, size_t num);
+
+    template<class iter_t>
+    void pop_move_front(iter_t it_dest, size_t num);
+
+    template<class iter_t>
+    void pop_copy_back(iter_t it_dest, size_t num);
+
+    template<class iter_t>
+    void pop_move_back(iter_t it_dest, size_t num);
+
+    template<class iter_t>
+    void push_copy_front(iter_t it_src, size_t num);
+
+    template<class iter_t>
+    void push_move_front(iter_t it_src, size_t num);
+
+    template<class iter_t>
+    void push_copy_back(iter_t it_src, size_t num);
+
+    template<class iter_t>
+    void push_move_back(iter_t it_src, size_t num);
+
+//---------------------------------------------------------------------------
+};//               End of class circular_buffer
+//---------------------------------------------------------------------------
+//
+//
+//############################################################################
+//                                                                          ##
+//             N O N    I N L I N E    F U N C T I O N S                    ##
+//                                                                          ##
+//############################################################################
+//
+//------------------------------------------------------------------------
+//  function : pop_copy_front
+/// @brief copy and delete num elements from the front of the buffer
+/// @param it_dest : iterator to the first position where copy the elements
+/// @param num : number of elements to copy
+//-----------------------------------------------------------------------
+template <class Value_t, uint32_t Power2>
+template<class iter_t>
+void circular_buffer<Value_t, Power2>
+::pop_copy_front(iter_t it_dest, size_t num)
+{
+    static_assert ( std::is_same <value_iter<iter_t>, Value_t>::value,
+                    "Incompatible iterator");
+    if (num == 0) return;
+#ifdef __BS_DEBUG
+    assert ( num <= nelem);
+#endif
+    nelem -= num;
+    size_t pos = first_pos;
+    first_pos = (first_pos + num) & MASK;
+    for (size_t i = 0; i < num; ++i)
+    {
+        *(it_dest++) = ptr[pos++ & MASK];
+    };
+    first_pos &= MASK;
+};
+//
+//------------------------------------------------------------------------
+//  function : pop_move_front
+/// @brief move num elements from the front of the buffer to the place
+//         pointed by it_dest
+/// @param it_dest : iterator to the first position where move the elements
+/// @param num : number of elements to move
+//-----------------------------------------------------------------------
+template <class Value_t, uint32_t Power2>
+template<class iter_t>
+void circular_buffer<Value_t, Power2>
+:: pop_move_front(iter_t it_dest, size_t num)
+{
+    static_assert ( std::is_same <value_iter<iter_t>, Value_t>::value,
+                    "Incompatible iterator");
+    if (num == 0) return;
+#ifdef __BS_DEBUG
+    assert ( num <= nelem);
+#endif
+    nelem -= num;
+    size_t pos = first_pos;
+    first_pos = (first_pos + num) & MASK;
+    for (size_t i = 0; i < num; ++i)
+    {
+        *(it_dest++) = std::move(ptr[pos++ & MASK]);
+    };
+    first_pos &= MASK;
+};
+//
+//------------------------------------------------------------------------
+//  function : pop_copy_back
+/// @brief copy and delete num elements from the back of the buffer
+/// @param p1 : iterator where begin to copy the elements
+/// @param num : number of elements to copy
+//-----------------------------------------------------------------------
+template <class Value_t, uint32_t Power2>
+template<class iter_t>
+void circular_buffer<Value_t, Power2>
+::pop_copy_back(iter_t it_dest, size_t num)
+{
+    static_assert ( std::is_same <value_iter<iter_t>, Value_t>::value,
+                    "Incompatible iterator");
+    if (num == 0) return;
+#ifdef __BS_DEBUG
+    assert ( num <= nelem);
+#endif
+    nelem -= num;
+    size_t pos = (first_pos + nelem) & MASK;
+    for (size_t i = 0; i < num; ++i)
+    {
+        *(it_dest++) = ptr[pos++ & MASK];
+    };
+};
+//
+//------------------------------------------------------------------------
+//  function : pop_move_back
+/// @brief move and delete num elements from the back of the buffer
+/// @param p1 : iterator where begin to move the elements
+/// @param num : number of elements to move
+//-----------------------------------------------------------------------
+template <class Value_t, uint32_t Power2>
+template<class iter_t>
+void circular_buffer<Value_t, Power2>
+::pop_move_back(iter_t it_dest, size_t num)
+{
+    static_assert ( std::is_same <value_iter<iter_t>, Value_t>::value,
+                    "Incompatible iterator");
+    if (num == 0) return;
+#ifdef __BS_DEBUG
+    assert ( num <= nelem);
+#endif
+    nelem -= num;
+    size_t pos = (first_pos + nelem) & MASK;
+    for (size_t i = 0; i < num; ++i)
+    {
+        *(it_dest++) = std::move(ptr[pos++ & MASK]);
+    };
+};
+//
+//------------------------------------------------------------------------
+//  function : push_copy_front
+/// @brief copy num elements in the front of the buffer
+/// @param it_src : iterator from where begin to copy the elements
+/// @param mun : number of element to copy
+//-----------------------------------------------------------------------
+template <class Value_t, uint32_t Power2>
+template<class iter_t>
+void circular_buffer<Value_t, Power2>
+::push_copy_front(iter_t it_src, size_t num)
+{
+    static_assert ( std::is_same <value_iter<iter_t>, Value_t>::value,
+                    "Incompatible iterator");
+    if (num == 0) return;
+#ifdef __BS_DEBUG
+    assert ( free_size() >= num);
+#endif
+    nelem += num;
+
+    first_pos = (first_pos + NMAX - num) & MASK;
+    size_t pos = first_pos;
+    for (size_t i = 0; i < num; ++i)
+    {
+        ptr[(pos++) & MASK] = *(it_src++);
+    };
+};
+//
+//------------------------------------------------------------------------
+//  function : push_move_front
+/// @brief move num elements in the front of the buffer
+/// @param p1 : iterator from where begin to move the elements
+/// @param mun : number of element to move
+//-----------------------------------------------------------------------
+template <class Value_t, uint32_t Power2>
+template<class iter_t>
+void circular_buffer<Value_t, Power2>
+::push_move_front(iter_t it_src, size_t num)
+{
+    static_assert ( std::is_same <value_iter<iter_t>, Value_t>::value,
+                    "Incompatible iterator");
+    if (num == 0) return;
+#ifdef __BS_DEBUG
+    assert ( free_size() >= num);
+#endif
+    nelem += num;
+    size_t pos = first_pos;
+    for (size_t i = 0; i < num; ++i)
+    {
+        ptr[(pos++) & MASK] = std::move(*(it_src++));
+    };
+};
+//
+//------------------------------------------------------------------------
+//  function : push_copy_back
+/// @brief copy num elements in the back of the buffer
+/// @param p1 : iterator from where begin to copy the elements
+/// @param mun : number of element to copy
+//-----------------------------------------------------------------------
+template <class Value_t, uint32_t Power2>
+template<class iter_t>
+void circular_buffer<Value_t, Power2>
+::push_copy_back(iter_t it_src, size_t num)
+{
+    static_assert ( std::is_same <value_iter<iter_t>, Value_t>::value,
+                    "Incompatible iterator");
+    if (num == 0) return;
+#ifdef __BS_DEBUG
+    assert ( free_size() >= num);
+#endif
+    size_t pos = first_pos + nelem;
+    nelem += num;
+    for (size_t i = 0; i < num; ++i)
+    {
+        ptr[(pos++) & MASK] = *(it_src++);
+    };
+};
+//
+//------------------------------------------------------------------------
+//  function : push_move_back
+/// @brief move num elements in the back of the buffer
+/// @param p1 : iterator from where begin to move the elements
+/// @param mun : number of element to move
+//-----------------------------------------------------------------------
+template <class Value_t, uint32_t Power2>
+template<class iter_t>
+void circular_buffer<Value_t, Power2>
+::push_move_back(iter_t it_src, size_t num)
+{
+    static_assert ( std::is_same <value_iter<iter_t>, Value_t>::value,
+                    "Incompatible iterator");
+    if (num == 0) return;
+#ifdef __BS_DEBUG
+    assert ( free_size() >= num);
+#endif
+    size_t pos = first_pos + nelem;
+    nelem += num;
+    for (size_t i = 0; i < num; ++i)
+    {
+        ptr[(pos++) & MASK] = std::move(*(it_src++));
+    };
+};
+
+//****************************************************************************
+};// End namespace util
+};// End namespace common
+};// End namespace sort
+};// End namespace boost
+//****************************************************************************
+#endif
diff --git a/vendor/boost/sort/common/util/insert.hpp b/vendor/boost/sort/common/util/insert.hpp
new file mode 100644
index 00000000..219fa8a3
--- /dev/null
+++ b/vendor/boost/sort/common/util/insert.hpp
@@ -0,0 +1,142 @@
+//----------------------------------------------------------------------------
+/// @file insert.hpp
+/// @brief
+///
+/// @author Copyright (c) 2016 Francisco José Tapia (fjtapia@gmail.com )\n
+///         Distributed under the Boost Software License, Version 1.0.\n
+///         ( See accompanying file LICENSE_1_0.txt or copy at
+///           http://www.boost.org/LICENSE_1_0.txt  )
+/// @version 0.1
+///
+/// @remarks
+//-----------------------------------------------------------------------------
+#ifndef __BOOST_SORT_COMMON_UTIL_INSERT_HPP
+#define __BOOST_SORT_COMMON_UTIL_INSERT_HPP
+
+//#include <boost/sort/spinsort/util/indirect.hpp>
+#include <boost/sort/common/util/insert.hpp>
+#include <boost/sort/common/util/traits.hpp>
+#include <boost/sort/common/util/algorithm.hpp>
+#include <cstdlib>
+#include <functional>
+#include <iterator>
+#include <memory>
+#include <type_traits>
+#include <vector>
+#include <cstddef>
+
+namespace boost
+{
+namespace sort
+{
+namespace common
+{
+namespace util
+{
+namespace here = boost::sort::common::util;
+//
+//############################################################################
+//
+//          D E F I N I T I O N S    O F    F U N C T I O N S
+//    
+// template < class Iter1_t, class Iter2_t, typename Compare>
+// void insert_sorted (Iter1_t first, Iter1_t mid, Iter1_t last,
+//                     Compare comp, Iter2_t  it_aux)
+//
+//############################################################################
+//
+//-----------------------------------------------------------------------------
+//  function : insert_sorted
+/// @brief : Insertion sort of elements sorted
+/// @param first: iterator to the first element of the range
+/// @param mid : last pointer of the sorted data, and first pointer to the
+///               elements to insert
+/// @param last : iterator to the next element of the last in the range
+/// @param comp :
+/// @comments : the two ranges are sorted and in it_aux there is spave for 
+///             to store temporally the elements to insert
+//-----------------------------------------------------------------------------
+template<class Iter1_t, class Iter2_t, typename Compare>
+static void insert_sorted(Iter1_t first, Iter1_t mid, Iter1_t last,
+                          Compare comp, Iter2_t it_aux)
+{
+    //------------------------------------------------------------------------
+    //                 metaprogram
+    //------------------------------------------------------------------------
+    typedef value_iter<Iter1_t> value_t;
+    typedef value_iter<Iter2_t> value2_t;
+    static_assert (std::is_same< value_t, value2_t>::value,
+                    "Incompatible iterators\n");
+
+    //--------------------------------------------------------------------
+    //                   program
+    //--------------------------------------------------------------------
+    if (mid == last) return;
+    if (first == mid) return;
+
+    //------------------------------------------------------------------------
+    // creation of the vector of elements to insert and their position in the
+    // sorted part
+    // the data are inserted in it_aux
+    //-----------------------------------------------------------------------
+    move_forward(it_aux, mid, last);
+
+    // search of the iterators where insert the new elements
+    size_t ndata = last - mid;
+    Iter1_t mv_first = mid, mv_last = mid;
+
+    for (size_t i = ndata; i > 0; --i)
+    {
+        mv_last = mv_first;
+        mv_first = std::upper_bound(first, mv_last, it_aux[i - 1], comp);
+        Iter1_t it1 = here::move_backward(mv_last + i, mv_first, mv_last);
+        *(it1 - 1) = std::move(it_aux[i - 1]);
+    };
+};
+
+template<class Iter1_t, class Iter2_t, typename Compare>
+static void insert_sorted_backward(Iter1_t first, Iter1_t mid, Iter1_t last,
+                                   Compare comp, Iter2_t it_aux)
+{
+    //------------------------------------------------------------------------
+    //                 metaprogram
+    //------------------------------------------------------------------------
+    typedef value_iter<Iter1_t> value_t;
+    typedef value_iter<Iter2_t> value2_t;
+    static_assert (std::is_same< value_t, value2_t>::value,
+                    "Incompatible iterators\n");
+
+    //--------------------------------------------------------------------
+    //                   program
+    //--------------------------------------------------------------------
+    if (mid == last) return;
+    if (first == mid) return;
+    //------------------------------------------------------------------------
+    // creation of the vector of elements to insert and their position in the
+    // sorted part
+    // the data are inserted in it_aux
+    //-----------------------------------------------------------------------
+    move_forward(it_aux, first, mid);
+
+    // search of the iterators where insert the new elements
+    size_t ndata = mid - first;
+    Iter1_t mv_first = mid, mv_last = mid;
+
+    for (size_t i = 0; i < ndata; ++i)
+    {
+        mv_first = mv_last;
+        mv_last = std::lower_bound(mv_first, last, it_aux[i], comp);
+        Iter1_t it1 = move_forward(mv_first - (ndata - i), mv_first, mv_last);
+        *(it1) = std::move(it_aux[i]);
+    };
+
+};
+//
+//****************************************************************************
+};//    End namespace util
+};//    End namepspace common
+};//    End namespace sort
+};//    End namepspace boost
+//****************************************************************************
+//
+#endif
diff --git a/vendor/boost/sort/common/util/merge.hpp b/vendor/boost/sort/common/util/merge.hpp
new file mode 100644
index 00000000..5fc90c0f
--- /dev/null
+++ b/vendor/boost/sort/common/util/merge.hpp
@@ -0,0 +1,494 @@
+//----------------------------------------------------------------------------
+/// @file merge.hpp
+/// @brief low level merge functions
+///
+/// @author Copyright (c) 2016 Francisco Jose Tapia (fjtapia@gmail.com )\n
+///         Distributed under the Boost Software License, Version 1.0.\n
+///         ( See accompanying file LICENSE_1_0.txt or copy at
+///           http://www.boost.org/LICENSE_1_0.txt  )
+/// @version 0.1
+///
+/// @remarks
+//-----------------------------------------------------------------------------
+#ifndef __BOOST_SORT_COMMON_UTIL_MERGE_HPP
+#define __BOOST_SORT_COMMON_UTIL_MERGE_HPP
+
+#include <algorithm>
+#include <functional>
+#include <iterator>
+#include <memory>
+
+#include <boost/sort/common/util/algorithm.hpp>
+#include <boost/sort/common/util/traits.hpp>
+#include <boost/sort/common/util/circular_buffer.hpp>
+
+namespace boost
+{
+namespace sort
+{
+namespace common
+{
+namespace util
+{
+namespace here = boost::sort::common::util;
+//----------------------------------------------------------------------------
+//
+//           F U N C T I O N S    I N    T H E     F I L E
+//----------------------------------------------------------------------------
+//
+// template < class Iter1_t, class Iter2_t, class Compare >
+// Iter2_t merge (Iter1_t buf1, const Iter1_t end_buf1, Iter1_t buf2,
+//                const Iter1_t end_buf2, Iter2_t buf_out, Compare comp)
+//
+// template < class Iter_t, class Value_t, class Compare >
+// Value_t *merge_construct (Iter_t first1, const Iter_t last1, Iter_t first2,
+//                           const Iter_t last2, Value_t *it_out, Compare comp)
+//
+// template < class Iter1_t, class Iter2_t, class Compare >
+// Iter2_t merge_half (Iter1_t buf1, const Iter1_t end_buf1, Iter2_t buf2,
+//                     const Iter2_t end_buf2, Iter2_t buf_out, Compare comp)
+//
+// template < class Iter1_t, class Iter2_t, class Compare >
+// Iter2_t merge_half_backward (Iter1_t buf1,  Iter1_t end_buf1,
+//                              Iter2_t buf2, Iter2_t end_buf2,
+//                              Iter1_t end_buf_out, Compare comp)
+//
+// template < class Iter1_t, class Iter2_t, class Iter3_t, class Compare >
+// bool merge_uncontiguous (Iter1_t src1, const Iter1_t end_src1,
+//                          Iter2_t src2, const Iter2_t end_src2,
+//                          Iter3_t aux, Compare comp)
+//
+// template < class Iter1_t, class Iter2_t, class Compare >
+// bool merge_contiguous (Iter1_t src1, Iter1_t src2, Iter1_t end_src2,
+//                        Iter2_t buf, Compare comp)
+//
+// template < class Iter_t, class Circular ,class Compare >
+// bool merge_circular  (Iter_t buf1, Iter_t end_buf1,
+//                       Iter_t buf2, Iter_t end_buf2,
+//                       Circular &circ, Compare comp, Iter_t &it_aux)
+//
+//----------------------------------------------------------------------------
+//
+//-----------------------------------------------------------------------------
+//  function : merge
+/// @brief Merge two contiguous buffers pointed by buf1 and buf2, and put
+///        in the buffer pointed by buf_out
+///
+/// @param buf1 : iterator to the first element in the first buffer
+/// @param end_buf1 : final iterator of first buffer
+/// @param buf2 : iterator to the first iterator to the second buffer
+/// @param end_buf2 : final iterator of the second buffer
+/// @param buf_out : buffer where move the elements merged
+/// @param comp : comparison object
+//-----------------------------------------------------------------------------
+template<class Iter1_t, class Iter2_t, class Iter3_t, class Compare>
+static Iter3_t merge(Iter1_t buf1, const Iter1_t end_buf1, Iter2_t buf2,
+                     const Iter2_t end_buf2, Iter3_t buf_out, Compare comp)
+{
+    //-------------------------------------------------------------------------
+    //                       Metaprogramming
+    //------------------------------------------------------------------------- 
+    typedef value_iter<Iter1_t> value1_t;
+    typedef value_iter<Iter2_t> value2_t;
+    typedef value_iter<Iter3_t> value3_t;
+    static_assert (std::is_same< value1_t, value2_t >::value,
+                    "Incompatible iterators\n");
+    static_assert (std::is_same< value3_t, value2_t >::value,
+                    "Incompatible iterators\n");
+
+    //-------------------------------------------------------------------------
+    //                       Code
+    //-------------------------------------------------------------------------
+    const size_t MIN_CHECK = 1024;
+
+    if (size_t((end_buf1 - buf1) + (end_buf2 - buf2)) >= MIN_CHECK)
+    {
+        if (buf1 == end_buf1) return move_forward(buf_out, buf2, end_buf2);
+        if (buf2 == end_buf2) return move_forward(buf_out, buf1, end_buf1);
+
+        if (not comp(*buf2, *(end_buf1 - 1)))
+        {
+            Iter3_t mid = move_forward(buf_out, buf1, end_buf1);
+            return move_forward(mid, buf2, end_buf2);
+        };
+
+        if (comp(*(end_buf2 - 1), *buf1))
+        {
+            Iter3_t mid = move_forward(buf_out, buf2, end_buf2);
+            return move_forward(mid, buf1, end_buf1);
+        };
+    };
+    while ((buf1 != end_buf1) and (buf2 != end_buf2))
+    {
+        *(buf_out++) = (not comp(*buf2, *buf1)) ?
+                        std::move(*(buf1++)) : std::move(*(buf2++));
+    };
+
+    return (buf1 == end_buf1) ?
+                    move_forward(buf_out, buf2, end_buf2) :
+                    move_forward(buf_out, buf1, end_buf1);
+}
+;
+//
+//-----------------------------------------------------------------------------
+//  function : merge_construct
+/// @brief Merge two contiguous buffers pointed by first1 and first2, and put
+///        in the uninitialized buffer pointed by it_out
+///
+/// @param first1 : iterator to the first element in the first buffer
+/// @param last1 : last iterator of the first buffer
+/// @param first2 : iterator to the first element to the second buffer
+/// @param last2 : final iterator of the second buffer
+/// @param it_out : uninitialized buffer where move the elements merged
+/// @param comp : comparison object
+//-----------------------------------------------------------------------------
+template<class Iter1_t, class Iter2_t, class Value_t, class Compare>
+static Value_t *merge_construct(Iter1_t first1, const Iter1_t last1,
+                                Iter2_t first2, const Iter2_t last2,
+                                Value_t *it_out, Compare comp)
+{
+    //-------------------------------------------------------------------------
+    //                       Metaprogramming
+    //------------------------------------------------------------------------- 
+    typedef value_iter<Iter1_t> type1;
+    typedef value_iter<Iter2_t> type2;
+    static_assert (std::is_same< Value_t, type1 >::value,
+                    "Incompatible iterators\n");
+    static_assert (std::is_same< Value_t, type2 >::value,
+                    "Incompatible iterators\n");
+
+    //-------------------------------------------------------------------------
+    //                       Code
+    //-------------------------------------------------------------------------
+    const size_t MIN_CHECK = 1024;
+
+    if (size_t((last1 - first1) + (last2 - first2)) >= MIN_CHECK)
+    {
+        if (first1 == last1) return move_construct(it_out, first2, last2);
+        if (first2 == last2) return move_construct(it_out, first1, last1);
+
+        if (not comp(*first2, *(last1 - 1)))
+        {
+            Value_t* mid = move_construct(it_out, first1, last1);
+            return move_construct(mid, first2, last2);
+        };
+
+        if (comp(*(last2 - 1), *first1))
+        {
+            Value_t* mid = move_construct(it_out, first2, last2);
+            return move_construct(mid, first1, last1);
+        };
+    };
+    while (first1 != last1 and first2 != last2)
+    {
+        construct_object((it_out++),
+                        (not comp(*first2, *first1)) ?
+                                        std::move(*(first1++)) :
+                                        std::move(*(first2++)));
+    };
+    return (first1 == last1) ?
+                    move_construct(it_out, first2, last2) :
+                    move_construct(it_out, first1, last1);
+};
+//
+//---------------------------------------------------------------------------
+//  function : merge_half
+/// @brief : Merge two buffers. The first buffer is in a separate memory.
+///          The second buffer have a empty space before buf2 of the same size
+///          than the (end_buf1 - buf1)
+///
+/// @param buf1 : iterator to the first element of the first buffer
+/// @param end_buf1 : iterator to the last element of the first buffer
+/// @param buf2 : iterator to the first element of the second buffer
+/// @param end_buf2 : iterator to the last element of the second buffer
+/// @param buf_out : iterator to the first element to the buffer where put
+///                  the result
+/// @param comp : object for Compare two elements of the type pointed
+///                by the Iter1_t and Iter2_t
+//---------------------------------------------------------------------------
+template<class Iter1_t, class Iter2_t, class Compare>
+static Iter2_t merge_half(Iter1_t buf1, const Iter1_t end_buf1, Iter2_t buf2,
+                          const Iter2_t end_buf2, Iter2_t buf_out, Compare comp)
+{
+    //-------------------------------------------------------------------------
+    //                         Metaprogramming
+    //------------------------------------------------------------------------- 
+    typedef value_iter<Iter1_t> value1_t;
+    typedef value_iter<Iter2_t> value2_t;
+    static_assert (std::is_same< value1_t, value2_t >::value,
+                    "Incompatible iterators\n");
+
+    //-------------------------------------------------------------------------
+    //                         Code
+    //-------------------------------------------------------------------------
+#ifdef __BS_DEBUG
+    assert ( (buf2 - buf_out) == ( end_buf1 - buf1));
+#endif
+    const size_t MIN_CHECK = 1024;
+
+    if (size_t((end_buf1 - buf1) + (end_buf2 - buf2)) >= MIN_CHECK)
+    {
+        if (buf1 == end_buf1) return end_buf2;
+        if (buf2 == end_buf2) return move_forward(buf_out, buf1, end_buf1);
+
+        if (not comp(*buf2, *(end_buf1 - 1)))
+        {
+            move_forward(buf_out, buf1, end_buf1);
+            return end_buf2;
+        };
+
+        if (comp(*(end_buf2 - 1), *buf1))
+        {
+            Iter2_t mid = move_forward(buf_out, buf2, end_buf2);
+            return move_forward(mid, buf1, end_buf1);
+        };
+    };
+    while ((buf1 != end_buf1) and (buf2 != end_buf2))
+    {
+        *(buf_out++) = (not comp(*buf2, *buf1)) ?
+                        std::move(*(buf1++)) : std::move(*(buf2++));
+    };
+    return (buf2 == end_buf2)? move_forward(buf_out, buf1, end_buf1) : end_buf2;
+};
+
+//
+//---------------------------------------------------------------------------
+//  function : merge_half_backward
+/// @brief : Merge two buffers. The first buffer is in a separate memory.
+///          The second buffer have a empty space before buf2 of the same size
+///          than the (end_buf1 - buf1)
+///
+/// @param buf1 : iterator to the first element of the first buffer
+/// @param end_buf1 : iterator to the last element of the first buffer
+/// @param buf2 : iterator to the first element of the second buffer
+/// @param end_buf2 : iterator to the last element of the second buffer
+/// @param buf_out : iterator to the first element to the buffer where put
+///                  the result
+/// @param comp : object for Compare two elements of the type pointed
+///                by the Iter1_t and Iter2_t
+//---------------------------------------------------------------------------
+template<class Iter1_t, class Iter2_t, class Compare>
+static Iter2_t merge_half_backward(Iter1_t buf1, Iter1_t end_buf1, Iter2_t buf2,
+                                   Iter2_t end_buf2, Iter1_t end_buf_out,
+                                   Compare comp)
+{
+    //-------------------------------------------------------------------------
+    //                         Metaprogramming
+    //-------------------------------------------------------------------------
+    typedef value_iter<Iter1_t> value1_t;
+    typedef value_iter<Iter2_t> value2_t;
+    static_assert (std::is_same< value1_t, value2_t >::value,
+                    "Incompatible iterators\n");
+
+    //-------------------------------------------------------------------------
+    //                         Code
+    //-------------------------------------------------------------------------
+#ifdef __BS_DEBUG
+    assert ((end_buf_out - end_buf1) == (end_buf2 - buf2) );
+#endif
+    const size_t MIN_CHECK = 1024;
+
+    if (size_t((end_buf1 - buf1) + (end_buf2 - buf2)) >= MIN_CHECK)
+    {
+        if (buf2 == end_buf2) return buf1;
+        if (buf1 == end_buf1)
+            return here::move_backward(end_buf_out, buf2, end_buf2);
+
+        if (not comp(*buf2, *(end_buf1 - 1)))
+        {
+            here::move_backward(end_buf_out, buf2, end_buf2);
+            return buf1;
+        };
+
+        if (comp(*(end_buf2 - 1), *buf1))
+        {
+            Iter1_t mid = here::move_backward(end_buf_out, buf1, end_buf1);
+            return here::move_backward(mid, buf2, end_buf2);
+        };
+    };
+    while ((buf1 != end_buf1) and (buf2 != end_buf2))
+    {
+        *(--end_buf_out) =
+                        (not comp(*(end_buf2 - 1), *(end_buf1 - 1))) ?
+                                        std::move(*(--end_buf2)):
+                                        std::move(*(--end_buf1));
+    };
+    return (buf1 == end_buf1) ?
+                    here::move_backward(end_buf_out, buf2, end_buf2) : buf1;
+};
+
+//
+//-----------------------------------------------------------------------------
+//  function : merge_uncontiguous
+/// @brief : merge two uncontiguous buffers, placing the results in the buffers
+///          Use an auxiliary buffer pointed by aux
+///
+/// @param src1 : iterator to the first element of the first buffer
+/// @param end_src1 : last iterator  of the first buffer
+/// @param src2 : iterator to the first element of the second buffer
+/// @param end_src2 : last iterator  of the second buffer
+/// @param aux  : iterator to the first element of the auxiliary buffer
+/// @param comp : object for to Compare elements
+/// @return true : not changes done,  false : changes in the buffers
+/// @remarks
+//-----------------------------------------------------------------------------
+template<class Iter1_t, class Iter2_t, class Iter3_t, class Compare>
+static bool merge_uncontiguous(Iter1_t src1, const Iter1_t end_src1,
+                               Iter2_t src2, const Iter2_t end_src2,
+                               Iter3_t aux, Compare comp)
+{
+    //-------------------------------------------------------------------------
+    //                    Metaprogramming
+    //------------------------------------------------------------------------- 
+    typedef value_iter<Iter1_t> type1;
+    typedef value_iter<Iter2_t> type2;
+    typedef value_iter<Iter3_t> type3;
+    static_assert (std::is_same< type1, type2 >::value,
+                    "Incompatible iterators\n");
+    static_assert (std::is_same< type3, type2 >::value,
+                    "Incompatible iterators\n");
+
+    //-------------------------------------------------------------------------
+    //                    Code
+    //-------------------------------------------------------------------------
+    if (src1 == end_src1 or src2 == end_src2
+                    or not comp(*src2, *(end_src1 - 1))) return true;
+
+    while (src1 != end_src1 and not comp(*src2, *src1))
+        ++src1;
+
+    Iter3_t const end_aux = aux + (end_src1 - src1);
+    Iter2_t src2_first = src2;
+    move_forward(aux, src1, end_src1);
+
+    while ((src1 != end_src1) and (src2 != end_src2))
+    {
+        *(src1++) = std::move((not comp(*src2, *aux)) ? *(aux++) : *(src2++));
+    }
+
+    if (src2 == end_src2)
+    {
+        while (src1 != end_src1)
+            *(src1++) = std::move(*(aux++));
+        move_forward(src2_first, aux, end_aux);
+    }
+    else
+    {
+        merge_half(aux, end_aux, src2, end_src2, src2_first, comp);
+    };
+    return false;
+};
+
+//
+//-----------------------------------------------------------------------------
+//  function : merge_contiguous
+/// @brief : merge two contiguous buffers,using an auxiliary buffer pointed
+///          by buf. The results are in src1 and src2
+///
+/// @param src1: iterator to the first position of the first buffer
+/// @param src2: final iterator of the first buffer and first iterator
+///              of the second buffer
+/// @param end_src2 : final iterator of the second buffer
+/// @param buf  : iterator to buffer used as auxiliary memory
+/// @param comp : object for to Compare elements
+/// @return true : not changes done,  false : changes in the buffers
+//-----------------------------------------------------------------------------
+template<class Iter1_t, class Iter2_t, class Compare>
+static bool merge_contiguous(Iter1_t src1, Iter1_t src2, Iter1_t end_src2,
+                             Iter2_t buf, Compare comp)
+{
+    //-------------------------------------------------------------------------
+    //                      Metaprogramming
+    //------------------------------------------------------------------------- 
+    typedef value_iter<Iter1_t> type1;
+    typedef value_iter<Iter2_t> type2;
+    static_assert (std::is_same< type1, type2 >::value,
+                    "Incompatible iterators\n");
+
+    //-------------------------------------------------------------------------
+    //                         Code
+    //-------------------------------------------------------------------------
+    if (src1 == src2 or src2 == end_src2 or not comp(*src2, *(src2 - 1)))
+        return true;
+
+    Iter1_t end_src1 = src2;
+    while (src1 != end_src1 and not comp(*src2, *src1))
+        ++src1;
+
+    if (src1 == end_src1) return false;
+
+    size_t nx = end_src1 - src1;
+    move_forward(buf, src1, end_src1);
+    merge_half(buf, buf + nx, src2, end_src2, src1, comp);
+    return false;
+};
+//
+//-----------------------------------------------------------------------------
+//  function : merge_circular
+/// @brief : merge two buffers,using a circular buffer
+///          This function don't check the parameters
+/// @param buf1: iterator to the first position of the first buffer
+/// @param end_buf1: iterator after the last element of the first buffer
+/// @param buf2: iterator to the first element of the secind buffer
+/// @param end_buf2: iterator to the first element of the secind buffer
+/// @param circ : circular buffer
+/// @param comp : comparison object
+/// @return true : finished buf1,  false : finished buf2
+/// @comments : be carefully because the iterators buf1 and buf2 are modified
+//-----------------------------------------------------------------------------
+template<class Iter1_t, class Iter2_t, class Circular, class Compare>
+static bool merge_circular(Iter1_t buf1, Iter1_t end_buf1, Iter2_t buf2,
+                           Iter2_t end_buf2, Circular &circ, Compare comp,
+                           Iter1_t &it1_out, Iter2_t &it2_out)
+{
+    //-------------------------------------------------------------------------
+    //                      Metaprogramming
+    //-------------------------------------------------------------------------
+    typedef value_iter<Iter1_t> type1;
+    typedef value_iter<Iter2_t> type2;
+    static_assert (std::is_same< type1, type2 >::value,
+                    "Incompatible iterators\n");
+    typedef typename Circular::value_t type3;
+    static_assert (std::is_same<type1, type3>::value,
+                    "Incompatible iterators\n");
+
+    //-------------------------------------------------------------------------
+    //                      Code
+    //-------------------------------------------------------------------------
+#ifdef __BS_DEBUG
+    assert ( circ.free_size() >= size_t ((end_buf1-buf1) + (end_buf2-buf2)));
+#endif
+
+    if (not comp(*buf2, *(end_buf1 - 1)))
+    {
+        circ.push_move_back(buf1, (end_buf1 - buf1));
+        it1_out = end_buf1;
+        it2_out = buf2;
+        return true;
+    };
+    if (comp(*(end_buf2 - 1), *buf1))
+    {
+        circ.push_move_back(buf2, (end_buf2 - buf2));
+        it1_out = buf1;
+        it2_out = end_buf2;
+        return false;
+    }
+    while (buf1 != end_buf1 and buf2 != end_buf2)
+    {
+        circ.push_back(comp(*buf2, *buf1) ? std::move(*(buf2++))
+                                          : std::move(*(buf1++)));
+    };
+    it2_out = buf2;
+    it1_out = buf1;
+    bool ret = (buf1 == end_buf1);
+    return ret;
+};
+//
+//****************************************************************************
+};//    End namespace util
+};//    End namespace common
+};//    End namespace sort
+};//    End namespace boost
+//****************************************************************************
+//
+#endif
diff --git a/vendor/boost/sort/common/util/traits.hpp b/vendor/boost/sort/common/util/traits.hpp
new file mode 100644
index 00000000..68e5cf03
--- /dev/null
+++ b/vendor/boost/sort/common/util/traits.hpp
@@ -0,0 +1,123 @@
+//----------------------------------------------------------------------------
+/// @file traits.hpp
+/// @brief this file contains the metaprogramming classes  compare_iter and
+///         enable_if_not_integral
+/// @author Copyright(c) 2016 Francisco Jose Tapia (fjtapia@gmail.com )\n
+///         Distributed under the Boost Software License, Version 1.0.\n
+///         ( See accompanying file LICENSE_1_0.txt or copy at
+///           http://www.boost.org/LICENSE_1_0.txt  )
+/// @version 0.1
+///
+//-----------------------------------------------------------------------------
+#ifndef __BOOST_SORT_COMMON_UTIL_TRAITS_HPP
+#define __BOOST_SORT_COMMON_UTIL_TRAITS_HPP
+
+#include <functional>
+#include <iterator>
+#include <type_traits>
+
+namespace boost
+{
+namespace sort
+{
+namespace common
+{
+namespace util
+{
+//----------------------------------------------------------------------------
+//                  USING SENTENCES
+//----------------------------------------------------------------------------
+using std::iterator_traits;
+
+//
+//---------------------------------------------------------------------------
+/// @class value_iter
+/// @brief From the iterator, obtain the type pointed by it
+/// @remarks The main utility of this, is simplify the default template
+///          parameter of comparison
+//---------------------------------------------------------------------------
+template<class iter_t>
+using value_iter = typename iterator_traits< iter_t >::value_type;
+//
+//---------------------------------------------------------------------------
+/// @class compare_iter
+/// @brief From the iterator, received as template parameter, obtain the type
+///        of the object pointed by the iterator, and with this define the
+///        std::less with this type obtained
+/// @remarks The main utility of this, is simplify the default template
+///          parameter of comparison
+//---------------------------------------------------------------------------
+template<class iter_t>
+using compare_iter =  std::less< value_iter< iter_t > >;
+
+//
+//---------------------------------------------------------------------------
+/// @class enable_if_not_integral
+/// @brief This is a SFINAE class for to detect if the third parameter in the
+///        invocation of the parallel sorting algorithms is an integer
+///        representing the number of threads to use or is a comparison object
+/// @remarks
+//---------------------------------------------------------------------------
+template<class T>
+using enable_if_not_integral =
+      typename std::enable_if< !std::is_integral< T >::value >::type;
+//
+//---------------------------------------------------------------------------
+/// @class enable_if_integral
+/// @brief This is a SFINAE class for to detect if the third parameter in the
+///        invocation of the parallel sorting algorithms is an integer
+///        representing the number of threads to use or is a comparison object
+/// @remarks
+//---------------------------------------------------------------------------
+template<class T>
+using enable_if_integral =
+      typename std::enable_if< std::is_integral< T >::value >::type;
+
+//
+//---------------------------------------------------------------------------
+/// @class enable_if_string
+/// @brief This is a SFINAE class for to detect if the parameter is a
+///        std::string for to apply specialized parameters in the invocation
+///        of the block_indirect_sort algorithm
+/// @remarks
+//---------------------------------------------------------------------------
+template<class T>
+using enable_if_string =
+      typename std::enable_if< std::is_same< T, std::string >::value >::type;
+
+//
+//---------------------------------------------------------------------------
+/// @class enable_if_not_string
+/// @brief This is a SFINAE class for to detect if the parameter is a
+///        std::string for to apply specialized parameters in the invocation
+///        of the block_indirect_sort algorithm
+/// @remarks
+//---------------------------------------------------------------------------
+template<class T>
+using enable_if_not_string =
+      typename std::enable_if<! std::is_same< T, std::string >::value >::type;
+
+//
+//---------------------------------------------------------------------------
+/// @class constructor
+/// @brief create a functor with the constructor of a class for to be invoked
+///        from a bind or a lambda
+/// @remarks
+//---------------------------------------------------------------------------
+template<class T>
+struct constructor
+{
+    template<class ... Args>
+    void operator()(Args && ... args)
+    {
+        T(std::forward<Args> (args) ...);
+    };
+};
+//
+//****************************************************************************
+};// End namespace util
+};// End namespace common
+};// End namespace sort
+};// End namespace boost
+//****************************************************************************
+#endif
diff --git a/vendor/boost/sort/insert_sort/insert_sort.hpp b/vendor/boost/sort/insert_sort/insert_sort.hpp
new file mode 100644
index 00000000..d40302ad
--- /dev/null
+++ b/vendor/boost/sort/insert_sort/insert_sort.hpp
@@ -0,0 +1,119 @@
+//----------------------------------------------------------------------------
+/// @file insert_sort.hpp
+/// @brief Insertion Sort algorithm
+///
+/// @author Copyright (c) 2016 Francisco Jose Tapia (fjtapia@gmail.com )\n
+///         Distributed under the Boost Software License, Version 1.0.\n
+///         ( See accompanying file LICENSE_1_0.txt or copy at
+///           http://www.boost.org/LICENSE_1_0.txt  )
+/// @version 0.1
+///
+/// @remarks
+//-----------------------------------------------------------------------------
+#ifndef __BOOST_SORT_INTROSORT_DETAIL_INSERT_SORT_HPP
+#define __BOOST_SORT_INTROSORT_DETAIL_INSERT_SORT_HPP
+
+#include <functional>
+#include <iterator>
+#include <algorithm>
+#include <utility> // std::swap
+#include <boost/sort/common/util/traits.hpp>
+#include <boost/sort/common/util/insert.hpp>
+
+namespace boost
+{
+namespace sort
+{
+using common::util::compare_iter;
+using common::util::value_iter;
+//
+//-----------------------------------------------------------------------------
+//  function : insert_sort
+/// @brief : Insertion sort algorithm
+/// @param first: iterator to the first element of the range
+/// @param last : iterator to the next element of the last in the range
+/// @param comp : object for to do the comparison between the elements
+/// @remarks This algorithm is O(N^2)
+//-----------------------------------------------------------------------------
+template < class Iter_t, typename Compare = compare_iter < Iter_t > >
+static void insert_sort (Iter_t first, Iter_t last,
+                         Compare comp = Compare())
+{
+    //--------------------------------------------------------------------
+    //                   DEFINITIONS
+    //--------------------------------------------------------------------
+    typedef value_iter< Iter_t > value_t;
+
+    if ((last - first) < 2) return;
+
+    for (Iter_t it_examine = first + 1; it_examine != last; ++it_examine)
+    {
+        value_t Aux = std::move (*it_examine);
+        Iter_t it_insertion = it_examine;
+
+        while (it_insertion != first and comp (Aux, *(it_insertion - 1)))
+        {
+            *it_insertion = std::move (*(it_insertion - 1));
+            --it_insertion;
+        };
+        *it_insertion = std::move (Aux);
+    };
+};
+
+/*
+//
+//-----------------------------------------------------------------------------
+//  function : insert_partial_sort
+/// @brief : Insertion sort of elements sorted
+/// @param first: iterator to the first element of the range
+/// @param mid : last pointer of the sorted data, and first pointer to the
+///               elements to insert
+/// @param last : iterator to the next element of the last in the range
+/// @param comp : object for to do the comparison between the elements
+/// @remarks This algorithm is O(N^2)
+//-----------------------------------------------------------------------------
+template < class Iter_t, typename Compare = compare_iter < Iter_t > >
+void insert_partial_sort (Iter_t first, Iter_t mid, Iter_t last,
+                          Compare comp = Compare())
+{
+    //--------------------------------------------------------------------
+    //                   DEFINITIONS
+    //--------------------------------------------------------------------
+    typedef value_iter< Iter_t > value_t;
+
+    if ( mid == last ) return ;
+    insert_sort ( mid, last, comp);
+    if (first == mid) return ;
+
+    // creation of the vector of elements to insert and their position in the
+    // sorted part
+    std::vector<Iter_t> viter ;
+    std::vector<value_t> vdata ;
+
+    for ( Iter_t alpha = mid ; alpha != last ; ++alpha)
+        vdata.push_back ( std::move ( *alpha));
+
+    Iter_t linf = first , lsup = mid ;
+    for ( uint32_t i= 0 ; i < vdata.size() ; ++i)
+    {   Iter_t it1 = std::upper_bound ( linf, lsup , vdata[i], comp);
+        viter.push_back ( it1 );
+        linf = it1 ;
+    };
+
+    // moving the elements
+    viter.push_back ( mid) ;
+    for ( uint32_t i = viter.size() -1 ; i!= 0 ; --i)
+    {   Iter_t src = viter[i], limit = viter[i-1];
+        Iter_t dest = src + ( i);
+        while ( src != limit) * (--dest) = std::move ( *(--src));
+        *(viter[i-1] + (i -1)) = std::move (vdata[i-1]);
+    };
+}
+*/
+//
+//****************************************************************************
+}; //    End namespace sort
+}; //    End namespace boost
+//****************************************************************************
+//
+#endif
diff --git a/vendor/boost/sort/parallel_stable_sort/parallel_stable_sort.hpp b/vendor/boost/sort/parallel_stable_sort/parallel_stable_sort.hpp
new file mode 100644
index 00000000..3dc92f21
--- /dev/null
+++ b/vendor/boost/sort/parallel_stable_sort/parallel_stable_sort.hpp
@@ -0,0 +1,292 @@
+//----------------------------------------------------------------------------
+/// @file parallel_stable_sort.hpp
+/// @brief This file contains the class parallel_stable_sort
+///
+/// @author Copyright (c) 2016 Francisco Jose Tapia (fjtapia@gmail.com )\n
+///         Distributed under the Boost Software License, Version 1.0.\n
+///         ( See accompanying file LICENSE_1_0.txt or copy at
+///           http://www.boost.org/LICENSE_1_0.txt  )
+/// @version 0.1
+///
+/// @remarks
+//-----------------------------------------------------------------------------
+#ifndef __BOOST_SORT_PARALLEL_DETAIL_PARALLEL_STABLE_SORT_HPP
+#define __BOOST_SORT_PARALLEL_DETAIL_PARALLEL_STABLE_SORT_HPP
+
+#include <boost/sort/sample_sort/sample_sort.hpp>
+#include <boost/sort/common/util/traits.hpp>
+#include <functional>
+#include <future>
+#include <iterator>
+#include <memory>
+#include <type_traits>
+#include <vector>
+
+namespace boost
+{
+namespace sort
+{
+namespace stable_detail
+{
+
+//---------------------------------------------------------------------------
+//                    USING SENTENCES
+//---------------------------------------------------------------------------
+namespace bsc = boost::sort::common;
+namespace bss = boost::sort::spin_detail;
+using bsc::range;
+using bsc::merge_half;
+using boost::sort::sample_detail::sample_sort;
+//
+///---------------------------------------------------------------------------
+/// @struct parallel_stable_sort
+/// @brief This a structure for to implement a parallel stable sort, exception
+///        safe
+//----------------------------------------------------------------------------
+template <class Iter_t, class Compare = compare_iter <Iter_t> >
+struct parallel_stable_sort
+{
+    //-------------------------------------------------------------------------
+    //                      DEFINITIONS
+    //-------------------------------------------------------------------------
+    typedef value_iter<Iter_t> value_t;
+
+    //-------------------------------------------------------------------------
+    //                     VARIABLES
+    //-------------------------------------------------------------------------
+    // Number of elements to sort
+    size_t nelem;
+    // Pointer to the auxiliary memory needed for the algorithm
+    value_t *ptr;
+    // Minimal number of elements for to be sorted in parallel mode
+    const size_t nelem_min = 1 << 16;
+
+    //------------------------------------------------------------------------
+    //                F U N C T I O N S
+    //------------------------------------------------------------------------
+    parallel_stable_sort (Iter_t first, Iter_t last)
+    : parallel_stable_sort (first, last, Compare(),
+                            std::thread::hardware_concurrency()) { };
+
+    parallel_stable_sort (Iter_t first, Iter_t last, Compare cmp)
+    : parallel_stable_sort (first, last, cmp,
+                            std::thread::hardware_concurrency()) { };
+
+    parallel_stable_sort (Iter_t first, Iter_t last, uint32_t num_thread)
+    : parallel_stable_sort (first, last, Compare(), num_thread) { };
+
+    parallel_stable_sort (Iter_t first, Iter_t last, Compare cmp,
+                          uint32_t num_thread);
+
+    //
+    //-----------------------------------------------------------------------------
+    //  function : destroy_all
+    /// @brief The utility is to destroy the temporary buffer used in the
+    ///        sorting process
+    //-----------------------------------------------------------------------------
+    void destroy_all()
+    {
+        if (ptr != nullptr) std::return_temporary_buffer(ptr);
+    };
+    //
+    //-----------------------------------------------------------------------------
+    //  function :~parallel_stable_sort
+    /// @brief destructor of the class. The utility is to destroy the temporary
+    ///        buffer used in the sorting process
+    //-----------------------------------------------------------------------------
+    ~parallel_stable_sort() {destroy_all(); } ;
+};
+// end struct parallel_stable_sort
+
+//
+//############################################################################
+//                                                                          ##
+//                                                                          ##
+//            N O N     I N L I N E      F U N C T I O N S                  ##
+//                                                                          ##
+//                                                                          ##
+//############################################################################
+//
+//-----------------------------------------------------------------------------
+//  function : parallel_stable_sort
+/// @brief constructor of the class
+///
+/// @param first : iterator to the first element of the range to sort
+/// @param last : iterator after the last element to the range to sort
+/// @param comp : object for to compare two elements pointed by Iter_t
+///                    iterators
+/// @param nthread : Number of threads to use in the process. When this value
+///                  is lower than 2, the sorting is done with 1 thread
+//-----------------------------------------------------------------------------
+template <class Iter_t, class Compare>
+parallel_stable_sort <Iter_t, Compare>
+::parallel_stable_sort (Iter_t first, Iter_t last, Compare comp,
+                        uint32_t nthread) : nelem(0), ptr(nullptr)
+{
+    range<Iter_t> range_initial(first, last);
+    assert(range_initial.valid());
+
+    nelem = range_initial.size();
+    size_t nptr = (nelem + 1) >> 1;
+
+    if (nelem < nelem_min or nthread < 2)
+    {
+        bss::spinsort<Iter_t, Compare>
+            (range_initial.first, range_initial.last, comp);
+        return;
+    };
+
+    //------------------- check if sort --------------------------------------
+    bool sw = true;
+    for (Iter_t it1 = first, it2 = first + 1;
+         it2 != last and (sw = not comp(*it2, *it1)); it1 = it2++);
+    if (sw) return;
+
+    //------------------- check if reverse sort ---------------------------
+    sw = true;
+    for (Iter_t it1 = first, it2 = first + 1;
+         it2 != last and (sw = comp(*it2, *it1)); it1 = it2++);
+    if (sw)
+    {
+        size_t nelem2 = nelem >> 1;
+        Iter_t it1 = first, it2 = last - 1;
+        for (size_t i = 0; i < nelem2; ++i)
+            std::swap(*(it1++), *(it2--));
+        return;
+    };
+
+    ptr = std::get_temporary_buffer<value_t>(nptr).first;
+    if (ptr == nullptr) throw std::bad_alloc();
+
+    //---------------------------------------------------------------------
+    //     Parallel Process
+    //---------------------------------------------------------------------
+    range<Iter_t> range_first(range_initial.first, range_initial.first + nptr);
+
+    range<Iter_t> range_second(range_initial.first + nptr, range_initial.last);
+
+    range<value_t *> range_buffer(ptr, ptr + nptr);
+
+    try
+    {
+        sample_sort<Iter_t, Compare>
+            (range_initial.first, range_initial.first + nptr,
+             comp, nthread, range_buffer);
+    } catch (std::bad_alloc &)
+    {
+        destroy_all();
+        throw std::bad_alloc();
+    };
+
+    try
+    {
+        sample_sort<Iter_t, Compare>
+            (range_initial.first + nptr,
+             range_initial.last, comp, nthread, range_buffer);
+    } catch (std::bad_alloc &)
+    {
+        destroy_all();
+        throw std::bad_alloc();
+    };
+
+    range_buffer = move_forward(range_buffer, range_first);
+    range_initial = merge_half(range_initial, range_buffer, range_second, comp);
+}; // end of constructor
+
+//
+//****************************************************************************
+};//    End namespace stable_detail
+//****************************************************************************
+//
+
+//---------------------------------------------------------------------------
+//                    USING SENTENCES
+//---------------------------------------------------------------------------
+namespace bsc = boost::sort::common;
+namespace bscu = bsc::util;
+namespace bss = boost::sort::spin_detail;
+using bsc::range;
+using bsc::merge_half;
+//
+//############################################################################
+//                                                                          ##
+//                                                                          ##
+//            P A R A L L E L _ S T A B L E _ S O R T                       ##
+//                                                                          ##
+//                                                                          ##
+//############################################################################
+//
+//-----------------------------------------------------------------------------
+//  function : parallel_stable_sort
+/// @brief : parallel stable sort with 2 parameters
+///
+/// @param first : iterator to the first element of the range to sort
+/// @param last : iterator after the last element to the range to sort
+//-----------------------------------------------------------------------------
+template<class Iter_t>
+void parallel_stable_sort(Iter_t first, Iter_t last)
+{
+    typedef bscu::compare_iter<Iter_t> Compare;
+    stable_detail::parallel_stable_sort<Iter_t, Compare>(first, last);
+};
+//
+//-----------------------------------------------------------------------------
+//  function : parallel_stable_sort
+/// @brief parallel stable sort with 3 parameters. The third is the number 
+///        of threads
+///
+/// @param first : iterator to the first element of the range to sort
+/// @param last : iterator after the last element to the range to sort
+/// @param nthread : Number of threads to use in the process. When this value
+///                  is lower than 2, the sorting is done with 1 thread
+//-----------------------------------------------------------------------------
+template<class Iter_t>
+void parallel_stable_sort(Iter_t first, Iter_t last, uint32_t nthread)
+{
+    typedef bscu::compare_iter<Iter_t> Compare;
+    stable_detail::parallel_stable_sort<Iter_t, Compare>(first, last, nthread);
+};
+//
+//-----------------------------------------------------------------------------
+//  function : parallel_stable_sort
+/// @brief : parallel stable sort with 3 parameters. The thisrd is the 
+///          comparison object
+///
+/// @param first : iterator to the first element of the range to sort
+/// @param last : iterator after the last element to the range to sort
+/// @param comp : object for to compare two elements pointed by Iter_t
+///               iterators
+//-----------------------------------------------------------------------------
+template <class Iter_t, class Compare,
+          bscu::enable_if_not_integral<Compare> * = nullptr>
+void parallel_stable_sort(Iter_t first, Iter_t last, Compare comp)
+{
+    stable_detail::parallel_stable_sort<Iter_t, Compare>(first, last, comp);
+};
+
+//
+//-----------------------------------------------------------------------------
+//  function : parallel_stable_sort
+/// @brief : parallel stable sort with 3 parameters. 
+///
+/// @param first : iterator to the first element of the range to sort
+/// @param last : iterator after the last element to the range to sort
+/// @param comp : object for to compare two elements pointed by Iter_t
+///               iterators
+/// @param nthread : Number of threads to use in the process. When this value
+///                  is lower than 2, the sorting is done with 1 thread
+//-----------------------------------------------------------------------------
+template<class Iter_t, class Compare>
+void parallel_stable_sort (Iter_t first, Iter_t last, Compare comp,
+                          uint32_t nthread)
+{
+    stable_detail::parallel_stable_sort<Iter_t, Compare>
+                                                (first, last, comp, nthread);
+}
+//
+//****************************************************************************
+};//    End namespace sort
+};//    End namespace boost
+//****************************************************************************
+//
+#endif
diff --git a/vendor/boost/sort/sample_sort/sample_sort.hpp b/vendor/boost/sort/sample_sort/sample_sort.hpp
new file mode 100644
index 00000000..ded1781c
--- /dev/null
+++ b/vendor/boost/sort/sample_sort/sample_sort.hpp
@@ -0,0 +1,560 @@
+//----------------------------------------------------------------------------
+/// @file sample_sort.hpp
+/// @brief contains the class sample_sort
+///
+/// @author Copyright (c) 2016 Francisco Jose Tapia (fjtapia@gmail.com )\n
+///         Distributed under the Boost Software License, Version 1.0.\n
+///         ( See accompanying file LICENSE_1_0.txt or copy at
+///           http://www.boost.org/LICENSE_1_0.txt  )
+/// @version 0.1
+///
+/// @remarks
+//-----------------------------------------------------------------------------
+#ifndef __BOOST_SORT_PARALLEL_DETAIL_SAMPLE_SORT_HPP
+#define __BOOST_SORT_PARALLEL_DETAIL_SAMPLE_SORT_HPP
+
+#include <functional>
+#include <future>
+#include <iterator>
+#include <memory>
+#include <type_traits>
+#include <vector>
+
+#include <algorithm>
+#include <boost/sort/spinsort/spinsort.hpp>
+#include <boost/sort/common/indirect.hpp>
+#include <boost/sort/common/util/atomic.hpp>
+#include <boost/sort/common/merge_four.hpp>
+#include <boost/sort/common/merge_vector.hpp>
+#include <boost/sort/common/range.hpp>
+
+namespace boost
+{
+namespace sort
+{
+namespace sample_detail
+{
+//---------------------------------------------------------------------------
+//                    USING SENTENCES
+//---------------------------------------------------------------------------
+namespace bsc = boost::sort::common;
+namespace bss = boost::sort::spin_detail;
+namespace bscu = boost::sort::common::util;
+using bsc::range;
+using bscu::atomic_add;
+using bsc::merge_vector4;
+using bsc::uninit_merge_level4;
+using bsc::less_ptr_no_null;
+
+//
+///---------------------------------------------------------------------------
+/// @struct sample_sort
+/// @brief This a structure for to implement a sample sort, exception
+///        safe
+/// @tparam
+/// @remarks
+//----------------------------------------------------------------------------
+template<class Iter_t, class Compare>
+struct sample_sort
+{
+    //------------------------------------------------------------------------
+    //                     DEFINITIONS
+    //------------------------------------------------------------------------
+    typedef value_iter<Iter_t> value_t;
+    typedef range<Iter_t> range_it;
+    typedef range<value_t *> range_buf;
+    typedef sample_sort<Iter_t, Compare> this_t;
+
+    //------------------------------------------------------------------------
+    //                VARIABLES AND CONSTANTS
+    //------------------------------------------------------------------------
+    // minimun numbers of elements for to be sortd in parallel mode
+    static const uint32_t thread_min = (1 << 16);
+
+    // Number of threads to use in the algorithm
+    // Number of intervals for to do the internal division of the data
+    uint32_t nthread, ninterval;
+
+    // Bool variables indicating if the auxiliary memory is constructed
+    // and indicating in the auxiliary memory had been obtained inside the
+    /// algorithm or had been received as a parameter
+    bool construct = false, owner = false;
+
+    // Comparison object for to compare two elements
+    Compare comp;
+
+    // Range with all the elements to sort
+    range_it global_range;
+
+    // range with the auxiliary memory
+    range_buf global_buf;
+
+    // vector of futures
+    std::vector<std::future<void>> vfuture;
+
+    // vector of vectors which contains the ranges to merge obtained in the
+    // subdivision
+    std::vector<std::vector<range_it>> vv_range_it;
+
+    // each vector of ranges of the vv_range_it, need their corresponding buffer
+    // for to do the merge
+    std::vector<std::vector<range_buf>> vv_range_buf;
+
+    // Initial vector of ranges
+    std::vector<range_it> vrange_it_ini;
+
+    // Initial vector of buffers
+    std::vector<range_buf> vrange_buf_ini;
+
+    // atomic counter for to know when are finished the function_t created
+    // inside a function
+    std::atomic<uint32_t> njob;
+
+    // Indicate if an error in the algorithm for to undo all
+    bool error;
+
+    //------------------------------------------------------------------------
+    //                       FUNCTIONS OF THE STRUCT
+    //------------------------------------------------------------------------
+    void initial_configuration(void);
+
+    sample_sort (Iter_t first, Iter_t last, Compare cmp, uint32_t num_thread,
+                 value_t *paux, size_t naux);
+
+    sample_sort(Iter_t first, Iter_t last)
+    : sample_sort (first, last, Compare(), std::thread::hardware_concurrency(),
+                   nullptr, 0) { };
+
+    sample_sort(Iter_t first, Iter_t last, Compare cmp)
+    : sample_sort(first, last, cmp, std::thread::hardware_concurrency(),
+                  nullptr, 0) { };
+
+    sample_sort(Iter_t first, Iter_t last, uint32_t num_thread)
+    : sample_sort(first, last, Compare(), num_thread, nullptr, 0) { };
+
+    sample_sort(Iter_t first, Iter_t last, Compare cmp, uint32_t num_thread)
+    : sample_sort(first, last, cmp, num_thread, nullptr, 0) { };
+
+    sample_sort(Iter_t first, Iter_t last, Compare cmp, uint32_t num_thread,
+                range_buf range_buf_initial)
+    : sample_sort(first, last, cmp, num_thread,
+                  range_buf_initial.first, range_buf_initial.size()) { };
+
+    void destroy_all(void);
+    //
+    //-----------------------------------------------------------------------------
+    //  function :~sample_sort
+    /// @brief destructor of the class. The utility is to destroy the temporary
+    ///        buffer used in the sorting process
+    //-----------------------------------------------------------------------------
+    ~sample_sort(void) { destroy_all(); };
+    //
+    //-----------------------------------------------------------------------
+    //  function : execute first
+    /// @brief this a function to assign to each thread in the first merge
+    //-----------------------------------------------------------------------
+    void execute_first(void)
+    {
+        uint32_t job = 0;
+        while ((job = atomic_add(njob, 1)) < ninterval)
+        {
+            uninit_merge_level4(vrange_buf_ini[job], vv_range_it[job],
+                            vv_range_buf[job], comp);
+        };
+    };
+    //
+    //-----------------------------------------------------------------------
+    //  function : execute
+    /// @brief this is a function to assignt each thread the final merge
+    //-----------------------------------------------------------------------
+    void execute(void)
+    {
+        uint32_t job = 0;
+        while ((job = atomic_add(njob, 1)) < ninterval)
+        {
+            merge_vector4(vrange_buf_ini[job], vrange_it_ini[job],
+                            vv_range_buf[job], vv_range_it[job], comp);
+        };
+    };
+    //
+    //-----------------------------------------------------------------------
+    //  function : first merge
+    /// @brief Implement the merge of the initially sparse ranges
+    //-----------------------------------------------------------------------
+    void first_merge(void)
+    { //---------------------------------- begin --------------------------
+        njob = 0;
+
+        for (uint32_t i = 0; i < nthread; ++i)
+        {
+            vfuture[i] = std::async(std::launch::async, &this_t::execute_first,
+                            this);
+        };
+        for (uint32_t i = 0; i < nthread; ++i)
+            vfuture[i].get();
+    };
+    //
+    //-----------------------------------------------------------------------
+    //  function : final merge
+    /// @brief Implement the final merge of the ranges
+    //-----------------------------------------------------------------------
+    void final_merge(void)
+    { //---------------------------------- begin --------------------------
+        njob = 0;
+
+        for (uint32_t i = 0; i < nthread; ++i)
+        {
+            vfuture[i] = std::async(std::launch::async, &this_t::execute, this);
+        };
+        for (uint32_t i = 0; i < nthread; ++i)
+            vfuture[i].get();
+    };
+    //----------------------------------------------------------------------------
+};
+//                    End class sample_sort
+//----------------------------------------------------------------------------
+//
+//############################################################################
+//                                                                          ##
+//              N O N    I N L I N E      F U N C T I O N S                 ##
+//                                                                          ##
+//                                                                          ##
+//############################################################################
+//
+//-----------------------------------------------------------------------------
+//  function : sample_sort
+/// @brief constructor of the class
+///
+/// @param first : iterator to the first element of the range to sort
+/// @param last : iterator after the last element to the range to sort
+/// @param cmp : object for to compare two elements pointed by Iter_t iterators
+/// @param num_thread : Number of threads to use in the process. When this value
+///                     is lower than 2, the sorting is done with 1 thread
+/// @param paux : pointer to the auxiliary memory. If nullptr, the memory is
+///               created inside the class
+/// @param naux : number of elements of the memory pointed by paux
+//-----------------------------------------------------------------------------
+template<class Iter_t, typename Compare>
+sample_sort<Iter_t, Compare>
+::sample_sort (Iter_t first, Iter_t last, Compare cmp, uint32_t num_thread,
+               value_t *paux, size_t naux)
+: nthread(num_thread), owner(false), comp(cmp), global_range(first, last),
+  global_buf(nullptr, nullptr), error(false)
+{
+    assert((last - first) >= 0);
+    size_t nelem = size_t(last - first);
+    construct = false;
+    njob = 0;
+    vfuture.resize(nthread);
+
+    // Adjust when have many threads and only a few elements
+    while (nelem > thread_min and (nthread * nthread) > (nelem >> 3))
+    {
+        nthread /= 2;
+    };
+    ninterval = (nthread << 3);
+
+    if (nthread < 2 or nelem <= (thread_min))
+    {
+        bss::spinsort<Iter_t, Compare>(first, last, comp);
+        return;
+    };
+
+    //------------------- check if sort --------------------------------------
+    bool sw = true;
+    for (Iter_t it1 = first, it2 = first + 1;
+                    it2 != last and (sw = not comp(*it2, *it1)); it1 = it2++);
+    if (sw) return;
+
+    //------------------- check if reverse sort ---------------------------
+    sw = true;
+    for (Iter_t it1 = first, it2 = first + 1;
+                    it2 != last and (sw = comp(*it2, *it1)); it1 = it2++);
+    if (sw)
+    {
+        size_t nelem2 = nelem >> 1;
+        Iter_t it1 = first, it2 = last - 1;
+        for (size_t i = 0; i < nelem2; ++i)
+            std::swap(*(it1++), *(it2--));
+        return;
+    };
+
+    if (paux != nullptr)
+    {
+        assert(naux != 0);
+        global_buf.first = paux;
+        global_buf.last = paux + naux;
+        owner = false;
+    }
+    else
+    {
+        value_t *ptr = std::get_temporary_buffer<value_t>(nelem).first;
+        if (ptr == nullptr) throw std::bad_alloc();
+        owner = true;
+        global_buf = range_buf(ptr, ptr + nelem);
+    };
+    //------------------------------------------------------------------------
+    //                    PROCESS
+    //------------------------------------------------------------------------
+    try
+    {
+        initial_configuration();
+    } catch (std::bad_alloc &)
+    {
+        error = true;
+    };
+    if (not error)
+    {
+        first_merge();
+        construct = true;
+        final_merge();
+    };
+    if (error)
+    {
+        destroy_all();
+        throw std::bad_alloc();
+    };
+}
+;
+//
+//-----------------------------------------------------------------------------
+//  function : destroy_all
+/// @brief destructor of the class. The utility is to destroy the temporary
+///        buffer used in the sorting process
+//-----------------------------------------------------------------------------
+template<class Iter_t, typename Compare>
+void sample_sort<Iter_t, Compare>::destroy_all(void)
+{
+    if (construct)
+    {
+        destroy(global_buf);
+        construct = false;
+    }
+    if (global_buf.first != nullptr and owner)
+        std::return_temporary_buffer(global_buf.first);
+}
+//
+//-----------------------------------------------------------------------------
+//  function : initial_configuration
+/// @brief Create the internal data structures, and obtain the inital set of
+///        ranges to merge
+//-----------------------------------------------------------------------------
+template<class Iter_t, typename Compare>
+void sample_sort<Iter_t, Compare>::initial_configuration(void)
+{
+    std::vector<range_it> vmem_thread;
+    std::vector<range_buf> vbuf_thread;
+    size_t nelem = global_range.size();
+
+    //------------------------------------------------------------------------
+    size_t cupo = nelem / nthread;
+    Iter_t it_first = global_range.first;
+    value_t *buf_first = global_buf.first;
+    vmem_thread.reserve(nthread + 1);
+    vbuf_thread.reserve(nthread + 1);
+
+    for (uint32_t i = 0; i < (nthread - 1); ++i, it_first += cupo, buf_first +=
+                    cupo)
+    {
+        vmem_thread.emplace_back(it_first, it_first + cupo);
+        vbuf_thread.emplace_back(buf_first, buf_first + cupo);
+    };
+
+    vmem_thread.emplace_back(it_first, global_range.last);
+    vbuf_thread.emplace_back(buf_first, global_buf.last);
+
+    //------------------------------------------------------------------------
+    // Sorting of the ranges
+    //------------------------------------------------------------------------
+    std::vector<std::future<void>> vfuture(nthread);
+
+    for (uint32_t i = 0; i < nthread; ++i)
+    {
+        auto func = [=]()
+        {
+            bss::spinsort<Iter_t, Compare> (vmem_thread[i].first,
+                            vmem_thread[i].last, comp,
+                            vbuf_thread[i]);
+        };
+        vfuture[i] = std::async(std::launch::async, func);
+    };
+
+    for (uint32_t i = 0; i < nthread; ++i)
+        vfuture[i].get();
+
+    //------------------------------------------------------------------------
+    // Obtain the vector of milestones
+    //------------------------------------------------------------------------
+    std::vector<Iter_t> vsample;
+    vsample.reserve(nthread * (ninterval - 1));
+
+    for (uint32_t i = 0; i < nthread; ++i)
+    {
+        size_t distance = vmem_thread[i].size() / ninterval;
+        for (size_t j = 1, pos = distance; j < ninterval; ++j, pos += distance)
+        {
+            vsample.push_back(vmem_thread[i].first + pos);
+        };
+    };
+    typedef less_ptr_no_null<Iter_t, Compare> compare_ptr;
+    typedef typename std::vector<Iter_t>::iterator it_to_it;
+
+    bss::spinsort<it_to_it, compare_ptr>(vsample.begin(), vsample.end(),
+                    compare_ptr(comp));
+
+    //------------------------------------------------------------------------
+    // Create the final milestone vector
+    //------------------------------------------------------------------------
+    std::vector<Iter_t> vmilestone;
+    vmilestone.reserve(ninterval);
+
+    for (uint32_t pos = nthread >> 1; pos < vsample.size(); pos += nthread)
+    {
+        vmilestone.push_back(vsample[pos]);
+    };
+
+    //------------------------------------------------------------------------
+    // Creation of the first vector of ranges
+    //------------------------------------------------------------------------
+    std::vector<std::vector<range<Iter_t>>>vv_range_first (nthread);
+
+    for (uint32_t i = 0; i < nthread; ++i)
+    {
+        Iter_t itaux = vmem_thread[i].first;
+
+        for (uint32_t k = 0; k < (ninterval - 1); ++k)
+        {
+            Iter_t it2 = std::upper_bound(itaux, vmem_thread[i].last,
+                            *vmilestone[k], comp);
+
+            vv_range_first[i].emplace_back(itaux, it2);
+            itaux = it2;
+        };
+        vv_range_first[i].emplace_back(itaux, vmem_thread[i].last);
+    };
+
+    //------------------------------------------------------------------------
+    // Copy in buffer and  creation of the final matrix of ranges
+    //------------------------------------------------------------------------
+    vv_range_it.resize(ninterval);
+    vv_range_buf.resize(ninterval);
+    vrange_it_ini.reserve(ninterval);
+    vrange_buf_ini.reserve(ninterval);
+
+    for (uint32_t i = 0; i < ninterval; ++i)
+    {
+        vv_range_it[i].reserve(nthread);
+        vv_range_buf[i].reserve(nthread);
+    };
+
+    Iter_t it = global_range.first;
+    value_t *it_buf = global_buf.first;
+
+    for (uint32_t k = 0; k < ninterval; ++k)
+    {
+        size_t nelem_interval = 0;
+
+        for (uint32_t i = 0; i < nthread; ++i)
+        {
+            size_t nelem_range = vv_range_first[i][k].size();
+            if (nelem_range != 0)
+            {
+                vv_range_it[k].push_back(vv_range_first[i][k]);
+            };
+            nelem_interval += nelem_range;
+        };
+
+        vrange_it_ini.emplace_back(it, it + nelem_interval);
+        vrange_buf_ini.emplace_back(it_buf, it_buf + nelem_interval);
+
+        it += nelem_interval;
+        it_buf += nelem_interval;
+    };
+}
+;
+//
+//****************************************************************************
+}
+;
+//    End namespace sample_detail
+//****************************************************************************
+//
+namespace bscu = boost::sort::common::util;
+//
+//############################################################################
+//                                                                          ##
+//                                                                          ##
+//                       S A M P L E _ S O R T                              ##
+//                                                                          ##
+//                                                                          ##
+//############################################################################
+//
+//-----------------------------------------------------------------------------
+//  function : sample_sort
+/// @brief parallel sample sort  algorithm (stable sort)
+///
+/// @param first : iterator to the first element of the range to sort
+/// @param last : iterator after the last element to the range to sort
+//-----------------------------------------------------------------------------
+template<class Iter_t>
+void sample_sort(Iter_t first, Iter_t last)
+{
+    typedef compare_iter<Iter_t> Compare;
+    sample_detail::sample_sort<Iter_t, Compare>(first, last);
+};
+//
+//-----------------------------------------------------------------------------
+//  function : sample_sort
+/// @brief parallel sample sort  algorithm (stable sort)
+///
+/// @param first : iterator to the first element of the range to sort
+/// @param last : iterator after the last element to the range to sort
+/// @param nthread : Number of threads to use in the process. When this value
+///                  is lower than 2, the sorting is done with 1 thread
+//-----------------------------------------------------------------------------
+template<class Iter_t>
+void sample_sort(Iter_t first, Iter_t last, uint32_t nthread)
+{
+    typedef compare_iter<Iter_t> Compare;
+    sample_detail::sample_sort<Iter_t, Compare>(first, last, nthread);
+};
+//
+//-----------------------------------------------------------------------------
+//  function : sample_sort
+/// @brief parallel sample sort  algorithm (stable sort)
+///
+/// @param first : iterator to the first element of the range to sort
+/// @param last : iterator after the last element to the range to sort
+/// @param comp : object for to compare two elements pointed by Iter_t
+///               iterators
+//-----------------------------------------------------------------------------
+template<class Iter_t, class Compare, bscu::enable_if_not_integral<Compare> * =
+                nullptr>
+void sample_sort(Iter_t first, Iter_t last, Compare comp)
+{
+    sample_detail::sample_sort<Iter_t, Compare>(first, last, comp);
+};
+//
+//-----------------------------------------------------------------------------
+//  function : sample_sort
+/// @brief parallel sample sort  algorithm (stable sort)
+///
+/// @param first : iterator to the first element of the range to sort
+/// @param last : iterator after the last element to the range to sort
+/// @param comp : object for to compare two elements pointed by Iter_t
+///               iterators
+/// @param nthread : Number of threads to use in the process. When this value
+///                  is lower than 2, the sorting is done with 1 thread
+//-----------------------------------------------------------------------------
+template<class Iter_t, class Compare>
+void sample_sort(Iter_t first, Iter_t last, Compare comp, uint32_t nthread)
+{
+    sample_detail::sample_sort<Iter_t, Compare>(first, last, comp, nthread);
+};
+//
+//****************************************************************************
+};//    End namespace sort
+};//    End namespace boost
+//****************************************************************************
+//
+#endif
diff --git a/vendor/boost/sort/spinsort/spinsort.hpp b/vendor/boost/sort/spinsort/spinsort.hpp
new file mode 100644
index 00000000..0e9f2d55
--- /dev/null
+++ b/vendor/boost/sort/spinsort/spinsort.hpp
@@ -0,0 +1,564 @@
+//----------------------------------------------------------------------------
+/// @file spinsort.hpp
+/// @brief Spin Sort algorithm
+///
+/// @author Copyright (c) 2016 Francisco José Tapia (fjtapia@gmail.com )\n
+///         Distributed under the Boost Software License, Version 1.0.\n
+///         ( See accompanying file LICENSE_1_0.txt or copy at
+///           http://www.boost.org/LICENSE_1_0.txt  )
+/// @version 0.1
+///
+/// @remarks
+//-----------------------------------------------------------------------------
+#ifndef __BOOST_SORT_PARALLEL_ALGORITHM_SPIN_SORT_HPP
+#define __BOOST_SORT_PARALLEL_ALGORITHM_SPIN_SORT_HPP
+
+//#include <boost/sort/spinsort/util/indirect.hpp>
+#include <boost/sort/insert_sort/insert_sort.hpp>
+#include <boost/sort/common/util/traits.hpp>
+#include <boost/sort/common/util/algorithm.hpp>
+#include <boost/sort/common/range.hpp>
+#include <boost/sort/common/indirect.hpp>
+#include <cstdlib>
+#include <functional>
+#include <iterator>
+#include <memory>
+#include <type_traits>
+#include <vector>
+#include <cstddef>
+
+namespace boost
+{
+namespace sort
+{
+namespace spin_detail
+{
+
+//----------------------------------------------------------------------------
+//                USING SENTENCES
+//----------------------------------------------------------------------------
+namespace bsc = boost::sort::common;
+using bsc::range;
+using bsc::util::nbits64;
+using bsc::util::compare_iter;
+using bsc::util::value_iter;
+using boost::sort::insert_sort;
+
+//
+//############################################################################
+//                                                                          ##
+//          D E F I N I T I O N S    O F    F U N C T I O N S               ##
+//                                                                          ##
+//############################################################################
+//
+template <class Iter1_t, class Iter2_t, typename Compare>
+static void insert_partial_sort (Iter1_t first, Iter1_t mid,
+                                 Iter1_t last, Compare comp,
+                                 const range<Iter2_t> &rng_aux);
+
+template<class Iter1_t, class Iter2_t, class Compare>
+static bool check_stable_sort (const range<Iter1_t> &rng_data,
+                               const range<Iter2_t> &rng_aux, Compare comp);
+
+template<class Iter1_t, class Iter2_t, class Compare>
+static void range_sort (const range<Iter1_t> &range1,
+                        const range<Iter2_t> &range2, Compare comp,
+                        uint32_t level);
+
+template<class Iter1_t, class Iter2_t, class Compare>
+static void sort_range_sort (const range<Iter1_t> &rng_data,
+                             const range<Iter2_t> &rng_aux, Compare comp);
+
+//
+//-----------------------------------------------------------------------------
+//  function : insert_partial_sort
+/// @brief : Insertion sort of elements sorted
+/// @param first: iterator to the first element of the range
+/// @param mid : last pointer of the sorted data, and first pointer to the
+///               elements to insert
+/// @param last : iterator to the next element of the last in the range
+/// @param comp :
+/// @comments : the two ranges are sorted
+//-----------------------------------------------------------------------------
+template<class Iter1_t, class Iter2_t, typename Compare>
+static void insert_partial_sort (Iter1_t first, Iter1_t mid, Iter1_t last,
+                                 Compare comp, const range<Iter2_t> &rng_aux)
+{
+    //------------------------------------------------------------------------
+    //                 metaprogram
+    //------------------------------------------------------------------------
+    typedef value_iter<Iter1_t> value_t;
+    typedef value_iter<Iter2_t> value2_t;
+    static_assert (std::is_same<value_t, value2_t>::value,
+                    "Incompatible iterators\n");
+
+    //--------------------------------------------------------------------
+    //                   program
+    //--------------------------------------------------------------------
+    assert(size_t(last - mid) <= rng_aux.size());
+
+    if (mid == last) return;
+    //insertionsort ( mid, last, comp);
+    if (first == mid) return;
+
+    //------------------------------------------------------------------------
+    // creation of the vector of elements to insert and their position in the
+    // sorted part
+    // the data are inserted in rng_aux
+    //-----------------------------------------------------------------------
+    std::vector<Iter1_t> viter;
+    Iter2_t beta = rng_aux.first, data = rng_aux.first;
+
+    for (Iter1_t alpha = mid; alpha != last; ++alpha)
+        *(beta++) = std::move(*alpha);
+
+    size_t ndata = last - mid;
+
+    Iter1_t linf = first, lsup = mid;
+    for (uint32_t i = 0; i < ndata; ++i)
+    {
+        Iter1_t it1 = std::upper_bound(linf, lsup, *(data + i), comp);
+        viter.push_back(it1);
+        linf = it1;
+    };
+
+    // moving the elements
+    viter.push_back(mid);
+    for (uint32_t i = viter.size() - 1; i != 0; --i)
+    {
+        Iter1_t src = viter[i], limit = viter[i - 1];
+        Iter1_t dest = src + (i);
+        while (src != limit) *(--dest) = std::move(*(--src));
+        *(viter[i - 1] + (i - 1)) = std::move(*(data + (i - 1)));
+    };
+}
+;
+//-----------------------------------------------------------------------------
+//  function : check_stable_sort
+/// @brief check if the elements between first and last are osted or reverse
+///        sorted. If the number of elements not sorted is small, insert in
+///        the sorted part
+/// @param range_input : range with the elements to sort
+/// @param range_buffer : range with the elements sorted
+/// @param comp : object for to compare two elements
+/// @param level : when is 1, sort with the insertionsort algorithm
+///                if not make a recursive call splitting the ranges
+//
+/// @comments : if the number of levels is odd, the data are in the first
+/// parameter of range_sort, and the results appear in the second parameter
+/// If the number of levels is even, the data are in the second
+/// parameter of range_sort, and the results are in the same parameter
+//-----------------------------------------------------------------------------
+template<class Iter1_t, class Iter2_t, class Compare>
+static bool check_stable_sort(const range<Iter1_t> &rng_data,
+                              const range<Iter2_t> &rng_aux, Compare comp)
+{
+    //------------------------------------------------------------------------
+    //              metaprogramming
+    //------------------------------------------------------------------------
+    typedef value_iter<Iter1_t> value_t;
+    typedef value_iter<Iter2_t> value2_t;
+    static_assert (std::is_same<value_t, value2_t>::value,
+                    "Incompatible iterators\n");
+
+    //------------------------------------------------------------------------
+    //                    program
+    //------------------------------------------------------------------------
+    // the maximun number of elements not ordered, for to be inserted in the
+    // sorted part
+    //const ptrdiff_t  min_insert_partial_sort = 32 ;
+    const size_t ndata = rng_data.size();
+    if (ndata < 32)
+    {
+        insert_sort(rng_data.first, rng_data.last, comp);
+        return true;
+    };
+    const size_t min_insert_partial_sort =
+                    ((ndata >> 3) < 33) ? 32 : (ndata >> 3);
+    if (ndata < 2) return true;
+
+    // check if sorted
+    bool sw = true;
+    Iter1_t it2 = rng_data.first + 1;
+    for (Iter1_t it1 = rng_data.first;
+                    it2 != rng_data.last and (sw = not comp(*it2, *it1)); it1 =
+                                    it2++)
+        ;
+    if (sw) return true;
+
+    // insert the elements between it1 and last
+    if (size_t(rng_data.last - it2) < min_insert_partial_sort)
+    {
+        sort_range_sort(range<Iter1_t>(it2, rng_data.last), rng_aux, comp);
+        insert_partial_sort(rng_data.first, it2, rng_data.last, comp, rng_aux);
+        return true;
+    };
+
+    // check if reverse sorted
+    if ((it2 != (rng_data.first + 1))) return false;
+    sw = true;
+    for (Iter1_t it1 = rng_data.first;
+                    it2 != rng_data.last and (sw = comp(*it2, *it1)); it1 =
+                                    it2++)
+        ;
+    if (size_t(rng_data.last - it2) >= min_insert_partial_sort) return false;
+
+    // reverse the elements between first and it1
+    size_t nreverse = it2 - rng_data.first;
+    Iter1_t alpha(rng_data.first), beta(it2 - 1), mid(
+                    rng_data.first + (nreverse >> 1));
+    while (alpha != mid)
+        std::swap(*(alpha++), *(beta--));
+
+    // insert the elements between it1 and last
+    if (it2 != rng_data.last)
+    {
+        sort_range_sort(range<Iter1_t>(it2, rng_data.last), rng_aux, comp);
+        insert_partial_sort(rng_data.first, it2, rng_data.last, comp, rng_aux);
+    };
+    return true;
+}
+;
+//-----------------------------------------------------------------------------
+//  function : range_sort
+/// @brief this function divide r_input in two parts, sort it,and merge moving
+///        the elements to range_buf
+/// @param range_input : range with the elements to sort
+/// @param range_buffer : range with the elements sorted
+/// @param comp : object for to compare two elements
+/// @param level : when is 1, sort with the insertionsort algorithm
+///                if not make a recursive call splitting the ranges
+//
+/// @comments : if the number of levels is odd, the data are in the first
+/// parameter of range_sort, and the results appear in the second parameter
+/// If the number of levels is even, the data are in the second
+/// parameter of range_sort, and the results are in the same parameter
+/// The two ranges must have the same size
+//-----------------------------------------------------------------------------
+template<class Iter1_t, class Iter2_t, class Compare>
+static void range_sort(const range<Iter1_t> &range1,
+                       const range<Iter2_t> &range2, Compare comp,
+                       uint32_t level)
+{
+    //-----------------------------------------------------------------------
+    //                  metaprogram
+    //-----------------------------------------------------------------------
+    typedef value_iter<Iter1_t> value_t;
+    typedef value_iter<Iter2_t> value2_t;
+    static_assert (std::is_same<value_t, value2_t>::value,
+                    "Incompatible iterators\n");
+
+    //-----------------------------------------------------------------------
+    //                  program
+    //-----------------------------------------------------------------------
+    typedef range<Iter1_t> range_it1;
+    typedef range<Iter2_t> range_it2;
+    assert(range1.size() == range2.size() and level != 0);
+
+    //------------------- check if sort --------------------------------------
+    if (range1.size() > 1024)
+    {
+        if ((level & 1) == 0)
+        {
+            if (check_stable_sort(range2, range1, comp)) return;
+        }
+        else
+        {
+            if (check_stable_sort(range1, range2, comp))
+            {
+                move_forward(range2, range1);
+                return;
+            };
+        };
+    };
+
+    //------------------- normal process -----------------------------------
+    size_t nelem1 = (range1.size() + 1) >> 1;
+    range_it1 range_input1(range1.first, range1.first + nelem1),
+                           range_input2(range1.first + nelem1, range1.last);
+
+    if (level < 2)
+    {
+        insert_sort(range_input1.first, range_input1.last, comp);
+        insert_sort(range_input2.first, range_input2.last, comp);
+    }
+    else
+    {
+        range_sort (range_it2(range2.first, range2.first + nelem1),
+                    range_input1, comp, level - 1);
+
+        range_sort (range_it2(range2.first + nelem1, range2.last),
+                    range_input2, comp, level - 1);
+    };
+
+    merge(range2, range_input1, range_input2, comp);
+}
+;
+//-----------------------------------------------------------------------------
+//  function : sort_range_sort
+/// @brief this sort elements using the range_sort function and receiving a
+///        buffer of initialized memory
+/// @param rng_data : range with the elements to sort
+/// @param rng_aux : range of at least the same memory than rng_data used as
+///                  auxiliary memory in the sorting
+/// @param comp : object for to compare two elements
+//-----------------------------------------------------------------------------
+template<class Iter1_t, class Iter2_t, class Compare>
+static void sort_range_sort(const range<Iter1_t> &rng_data,
+                            const range<Iter2_t> &rng_aux, Compare comp)
+{
+    //-----------------------------------------------------------------------
+    //                  metaprogram
+    //-----------------------------------------------------------------------
+    typedef value_iter<Iter1_t> value_t;
+    typedef value_iter<Iter2_t> value2_t;
+    static_assert (std::is_same<value_t, value2_t>::value,
+                    "Incompatible iterators\n");
+
+    //------------------------------------------------------------------------
+    //                    program
+    //------------------------------------------------------------------------
+    // minimal number of element before to jump to insertionsort
+    static const uint32_t sort_min = 32;
+    if (rng_data.size() <= sort_min)
+    {
+        insert_sort(rng_data.first, rng_data.last, comp);
+        return;
+    };
+
+#ifdef __BS_DEBUG
+    assert (rng_aux.size () >= rng_data.size ());
+#endif
+
+    range<Iter2_t> rng_buffer(rng_aux.first, rng_aux.first + rng_data.size());
+    uint32_t nlevel =
+                    nbits64(((rng_data.size() + sort_min - 1) / sort_min) - 1);
+    //assert (nlevel != 0);
+
+    if ((nlevel & 1) == 0)
+    {
+        range_sort(rng_buffer, rng_data, comp, nlevel);
+    }
+    else
+    {
+        range_sort(rng_data, rng_buffer, comp, nlevel);
+        move_forward(rng_data, rng_buffer);
+    };
+}
+;
+//
+//############################################################################
+//                                                                          ##
+//                              S T R U C T                                 ##
+//                                                                          ##
+//                           S P I N _ S O R T                              ##
+//                                                                          ##
+//############################################################################
+//---------------------------------------------------------------------------
+/// @struct spin_sort
+/// @brief  This class implement s stable sort algorithm with 1 thread, with
+///         an auxiliary memory of N/2 elements
+//----------------------------------------------------------------------------
+template<class Iter_t, typename Compare = compare_iter<Iter_t>>
+class spinsort
+{
+    //------------------------------------------------------------------------
+    //               DEFINITIONS AND CONSTANTS
+    //------------------------------------------------------------------------
+    typedef value_iter<Iter_t> value_t;
+    typedef range<Iter_t> range_it;
+    typedef range<value_t *> range_buf;
+    // When the number of elements to sort is smaller than Sort_min, are sorted
+    // by the insertion sort algorithm
+    static const uint32_t Sort_min = 36;
+
+    //------------------------------------------------------------------------
+    //                      VARIABLES
+    //------------------------------------------------------------------------
+    // Pointer to the auxiliary memory
+    value_t *ptr;
+
+    // Number of elements in the auxiliary memory
+    size_t nptr;
+
+    // construct indicate if the auxiliary memory in initialized, and owner
+    // indicate if the auxiliary memory had been created inside the object or
+    // had
+    // been received as a parameter
+    bool construct = false, owner = false;
+
+    //------------------------------------------------------------------------
+    //                   PRIVATE FUNCTIONS
+    //-------------------------------------------------------------------------
+    spinsort (Iter_t first, Iter_t last, Compare comp, value_t *paux,
+               size_t naux);
+
+public:
+    //------------------------------------------------------------------------
+    //                   PUBLIC FUNCTIONS
+    //-------------------------------------------------------------------------
+    spinsort(Iter_t first, Iter_t last, Compare comp = Compare())
+    : spinsort(first, last, comp, nullptr, 0) { };
+
+    spinsort(Iter_t first, Iter_t last, Compare comp, range_buf range_aux)
+    : spinsort(first, last, comp, range_aux.first, range_aux.size()) { };
+    //
+    //-----------------------------------------------------------------------
+    //  function :~spinsort
+    /// @brief destructor of the struct. Destroy the elements if construct is
+    /// true,
+    ///        and return the memory if owner is true
+    //-----------------------------------------------------------------------
+    ~spinsort(void)
+    {
+        if (construct)
+        {
+            destroy(range<value_t *>(ptr, ptr + nptr));
+            construct = false;
+        };
+        if (owner and ptr != nullptr) std::return_temporary_buffer(ptr);
+    };
+};
+//----------------------------------------------------------------------------
+//        End of class spinsort
+//----------------------------------------------------------------------------
+//
+//-------------------------------------------------------------------------
+//  function : spinsort
+/// @brief constructor of the struct
+//
+/// @param first : iterator to the first element of the range to sort
+/// @param last : iterator after the last element to the range to sort
+/// @param comp : object for to compare two elements pointed by Iter_t
+///               iterators
+/// @param paux : pointer to the auxiliary memory provided. If nullptr, the
+///               memory is created inside the class
+/// @param naux : number of elements pointed by paux
+//------------------------------------------------------------------------
+template <class Iter_t, typename Compare>
+spinsort <Iter_t, Compare>
+::spinsort (Iter_t first, Iter_t last, Compare comp, value_t *paux, size_t naux)
+: ptr(paux), nptr(naux), construct(false), owner(false)
+{
+    range<Iter_t> range_input(first, last);
+    assert(range_input.valid());
+
+    size_t nelem = range_input.size();
+    owner = construct = false;
+
+    nptr = (nelem + 1) >> 1;
+    size_t nelem_1 = nptr;
+    size_t nelem_2 = nelem - nelem_1;
+
+    if (nelem <= (Sort_min << 1))
+    {
+        insert_sort(range_input.first, range_input.last, comp);
+        return;
+    };
+
+    //------------------- check if sort ---------------------------------
+    bool sw = true;
+    for (Iter_t it1 = first, it2 = first + 1; it2 != last
+         and (sw = not comp(*it2, *it1)); it1 = it2++) ;
+    if (sw) return;
+
+    //------------------- check if reverse sort -------------------------
+    sw = true;
+    for (Iter_t it1 = first, it2 = first + 1;
+         it2 != last and (sw = comp(*it2, *it1)); it1 = it2++);
+    if (sw)
+    {
+        size_t nelem2 = nelem >> 1;
+        Iter_t it1 = first, it2 = last - 1;
+        for (size_t i = 0; i < nelem2; ++i)
+            std::swap(*(it1++), *(it2--));
+        return;
+    };
+
+    if (ptr == nullptr)
+    {
+        ptr = std::get_temporary_buffer<value_t>(nptr).first;
+        if (ptr == nullptr) throw std::bad_alloc();
+        owner = true;
+    };
+    range_buf range_aux(ptr, (ptr + nptr));
+
+    //---------------------------------------------------------------------
+    //                  Process
+    //---------------------------------------------------------------------
+    uint32_t nlevel = nbits64(((nelem + Sort_min - 1) / Sort_min) - 1) - 1;
+    assert(nlevel != 0);
+
+    if ((nlevel & 1) == 1)
+    {
+        //----------------------------------------------------------------
+        // if the number of levels is odd, the data are in the first
+        // parameter of range_sort, and the results appear in the second
+        // parameter
+        //----------------------------------------------------------------
+        range_it range_1(first, first + nelem_2), range_2(first + nelem_2,
+                        last);
+        range_aux = move_construct(range_aux, range_2);
+        construct = true;
+
+        range_sort(range_aux, range_2, comp, nlevel);
+        range_buf rng_bx(range_aux.first, range_aux.first + nelem_2);
+
+        range_sort(range_1, rng_bx, comp, nlevel);
+        merge_half(range_input, rng_bx, range_2, comp);
+    }
+    else
+    {
+        //----------------------------------------------------------------
+        // If the number of levels is even, the data are in the second
+        // parameter of range_sort, and the results are in the same
+        //  parameter
+        //----------------------------------------------------------------
+        range_it range_1(first, first + nelem_1), range_2(first + nelem_1,
+                        last);
+        range_aux = move_construct(range_aux, range_1);
+        construct = true;
+
+        range_sort(range_1, range_aux, comp, nlevel);
+
+        range_1.last = range_1.first + range_2.size();
+        range_sort(range_1, range_2, comp, nlevel);
+        merge_half(range_input, range_aux, range_2, comp);
+    };
+};
+
+//****************************************************************************
+};//    End namepspace spin_detail
+//****************************************************************************
+//
+namespace bsc = boost::sort::common;
+//-----------------------------------------------------------------------------
+//  function : spinsort
+/// @brief this function implement a single thread stable sort
+///
+/// @param first : iterator to the first element of the range to sort
+/// @param last : iterator after the last element to the range to sort
+/// @param comp : object for to compare two elements pointed by Iter_t
+///               iterators
+//-----------------------------------------------------------------------------
+template <class Iter_t, class Compare = compare_iter<Iter_t>>
+inline void spinsort (Iter_t first, Iter_t last, Compare comp = Compare())
+{
+    spin_detail::spinsort <Iter_t, Compare> (first, last, comp);
+};
+
+template <class Iter_t, class Compare = compare_iter<Iter_t>>
+inline void indirect_spinsort (Iter_t first, Iter_t last,
+                               Compare comp = Compare())
+{
+    typedef typename std::vector<Iter_t>::iterator itx_iter;
+    typedef common::less_ptr_no_null <Iter_t, Compare> itx_comp;
+    common::indirect_sort (spinsort<itx_iter, itx_comp>, first, last, comp);
+};
+
+//****************************************************************************
+};//    End namespace sort
+};//    End namepspace boost
+//****************************************************************************
+//
+#endif


Test patch:
diff --git a/docker2/test b/docker2/test
new file mode 100755
index 00000000..e70d5c2b
--- /dev/null
+++ b/docker2/test
@@ -0,0 +1,11 @@
+#!/usr/bin/env bash
+set -ex
+
+HERE=$(dirname $0)
+. $HERE/common
+
+# In case we switch agents between steps
+[ ! -z $(docker images -q $TAG_SHA) ] || docker pull $TAG_SHA
+
+## Just check that we can bring up the container and run something
+docker run -it -w /src --rm $TAG_SHA poppunk --version
diff --git a/test/.gitignore b/test/.gitignore
new file mode 100644
index 00000000..98c06456
--- /dev/null
+++ b/test/.gitignore
@@ -0,0 +1,1 @@
+*.fa
diff --git a/test/clean_test.py b/test/clean_test.py
index b1923144..1acd576e 100755
--- a/test/clean_test.py
+++ b/test/clean_test.py
@@ -12,10 +12,10 @@ def deleteDir(dirname):
         shutil.rmtree(dirname)
 
 sys.stderr.write("Cleaning up tests\n")
-refs = []
+dirty_files = ['example_db.info.csv']
 with open("references.txt", 'r') as ref_file:
     for line in ref_file:
-        refs.append(line.rstrip().split("\t")[1])
+        dirty_files.append(line.rstrip().split("\t")[1])
 
 # clean up
 outputDirs = [
@@ -37,8 +37,8 @@ def deleteDir(dirname):
     "example_viz_query_lineages",
     "example_mst",
     "example_sparse_mst",
-    "example_tsne",
-    "example_prune",
+    "example_mandrake",
+    "example_iterate",
     "example_refs",
     "example_api",
     "batch1",
@@ -50,7 +50,7 @@ def deleteDir(dirname):
 for outDir in outputDirs:
     deleteDir(outDir)
 
-for ref in refs:
+for ref in dirty_files:
     if os.path.isfile(ref):
         os.remove(ref)
 
diff --git a/test/inref_query.txt b/test/inref_query.txt
new file mode 100644
index 00000000..df782440
--- /dev/null
+++ b/test/inref_query.txt
@@ -0,0 +1,1 @@
+12673_8#43	12673_8#43.contigs_velvet.fa
diff --git a/test/manual.txt b/test/manual.txt
index 85c73b72..55fc76d5 100644
--- a/test/manual.txt
+++ b/test/manual.txt
@@ -1,2 +1,3 @@
 start 0,0
-end 0.3,0.3
+end 0.01,0.2
+scaled False
\ No newline at end of file
diff --git a/test/more_queries.txt b/test/more_queries.txt
new file mode 100644
index 00000000..a77133d8
--- /dev/null
+++ b/test/more_queries.txt
@@ -0,0 +1,6 @@
+12673_8_43	12673_8#43.contigs_velvet.fa
+19183_4_69	19183_4#69.contigs_velvet.fa
+12754_5_76	12754_5#76.contigs_velvet.fa
+12754_5_57	12754_5#57.contigs_velvet.fa
+19183_4_59	19183_4#59.contigs_velvet.fa
+12754_4_89_partial	12754_4#89.partial.contigs_velvet.fa
diff --git a/test/run_test.py b/test/run_test.py
index 2b66b5f5..4030c699 100755
--- a/test/run_test.py
+++ b/test/run_test.py
@@ -17,17 +17,13 @@
 else:
     python_cmd = "python"
 
-#easy run
 sys.stderr.write("Running database creation (--create-db)\n")
-subprocess.run(python_cmd + " ../poppunk-runner.py --create-db --r-files references.txt --min-k 13 --k-step 3 --output example_db --qc-filter prune --overwrite", shell=True, check=True)
+subprocess.run(python_cmd + " ../poppunk-runner.py --create-db --r-files references.txt --min-k 13 --k-step 3 --plot-fit 5 --output example_db --overwrite", shell=True, check=True)
 
 # create database with different QC options
-sys.stderr.write("Running database QC test (--create-db)\n")
-subprocess.run(python_cmd + " ../poppunk-runner.py --create-db --r-files references.txt --min-k 13 --k-step 3 --output example_qc --qc-filter continue --length-range 2000000 3000000 --overwrite", shell=True, check=True)
-
-# test updating order is correct
-sys.stderr.write("Running distance matrix order check (--update-db)\n")
-subprocess.run(python_cmd + " test-update.py", shell=True, check=True)
+sys.stderr.write("Running database QC test (--qc-db)\n")
+subprocess.run(python_cmd + " ../poppunk-runner.py --qc-db --ref-db example_db --type-isolate \"12754_4#79\" --overwrite", shell=True, check=True)
+subprocess.run(python_cmd + " ../poppunk-runner.py --qc-db --ref-db example_db --output example_qc --type-isolate \"12754_4#79\" --length-range 2000000 3000000 --overwrite", shell=True, check=True)
 
 #fit GMM
 sys.stderr.write("Running GMM model fit (--fit-model gmm)\n")
@@ -39,18 +35,18 @@
 
 #refine model with GMM
 sys.stderr.write("Running model refinement (--fit-model refine)\n")
-subprocess.run(python_cmd + " ../poppunk-runner.py --fit-model refine --ref-db example_db --output example_refine --neg-shift 0.2 --overwrite", shell=True, check=True)
+subprocess.run(python_cmd + " ../poppunk-runner.py --fit-model refine --ref-db example_db --output example_refine --neg-shift 0.15 --overwrite", shell=True, check=True)
 subprocess.run(python_cmd + " ../poppunk-runner.py --fit-model refine --ref-db example_db --output example_refine --manual-start manual.txt --overwrite", shell=True, check=True)
-subprocess.run(python_cmd + " ../poppunk-runner.py --fit-model refine --ref-db example_db --output example_refine --neg-shift 0.2 --overwrite --indiv-refine both", shell=True, check=True)
-subprocess.run(python_cmd + " ../poppunk-runner.py --fit-model refine --ref-db example_db --output example_refine --neg-shift 0.2 --overwrite --indiv-refine both --no-local", shell=True, check=True)
-subprocess.run(python_cmd + " ../poppunk-runner.py --fit-model refine --ref-db example_db --output example_refine --neg-shift 0.2 --overwrite --unconstrained", shell=True, check=True)
-subprocess.run(python_cmd + " ../poppunk-runner.py --fit-model refine --ref-db example_db --output example_refine --neg-shift 0.2 --overwrite --score-idx 1", shell=True, check=True)
-subprocess.run(python_cmd + " ../poppunk-runner.py --fit-model refine --ref-db example_db --output example_refine --neg-shift 0.2 --overwrite --score-idx 2", shell=True, check=True)
+subprocess.run(python_cmd + " ../poppunk-runner.py --fit-model refine --ref-db example_db --output example_refine --neg-shift 0.15 --overwrite --indiv-refine both", shell=True, check=True)
+subprocess.run(python_cmd + " ../poppunk-runner.py --fit-model refine --ref-db example_db --output example_refine --neg-shift 0.15 --overwrite --indiv-refine both --no-local", shell=True, check=True)
+subprocess.run(python_cmd + " ../poppunk-runner.py --fit-model refine --ref-db example_db --output example_refine --neg-shift 0.15 --overwrite --unconstrained", shell=True, check=True)
+subprocess.run(python_cmd + " ../poppunk-runner.py --fit-model refine --ref-db example_db --output example_refine --neg-shift 0.15 --overwrite --score-idx 1", shell=True, check=True)
+subprocess.run(python_cmd + " ../poppunk-runner.py --fit-model refine --ref-db example_db --output example_refine --neg-shift 0.15 --overwrite --score-idx 2", shell=True, check=True)
 subprocess.run(python_cmd + " ../poppunk-runner.py --fit-model threshold --threshold 0.003 --ref-db example_db --output example_threshold", shell=True, check=True)
 
 sys.stderr.write("Running multi boundary refinement (--multi-boundary and poppunk_iterate.py)\n")
-subprocess.run(python_cmd + " ../poppunk-runner.py --fit-model refine --ref-db example_db --output example_refine --neg-shift 0.2 --overwrite --multi-boundary 10", shell=True, check=True)
-subprocess.run(python_cmd + " ../scripts/poppunk_iterate.py --db example_refine --h5 example_db/example_db", shell=True, check=True)
+subprocess.run(python_cmd + " ../poppunk-runner.py --fit-model refine --ref-db example_db --output example_iterate --neg-shift -0.2 --overwrite --multi-boundary 10", shell=True, check=True)
+subprocess.run(python_cmd + " ../scripts/poppunk_iterate.py --db example_iterate --h5 example_db/example_db", shell=True, check=True)
 
 # lineage clustering
 sys.stderr.write("Running lineage clustering test (--fit-model lineage)\n")
@@ -64,13 +60,23 @@
 sys.stderr.write("Testing C++ extension\n")
 subprocess.run(python_cmd + " test-refine.py", shell=True, check=True)
 
-#assign query
+# assign query
 sys.stderr.write("Running query assignment\n")
 subprocess.run(python_cmd + " ../poppunk_assign-runner.py --query some_queries.txt --db example_db --model-dir example_refine --output example_query --overwrite --core --accessory", shell=True, check=True)
+subprocess.run(python_cmd + " ../poppunk_assign-runner.py --query some_queries.txt --db example_db --model-dir example_refine --output example_query --serial --overwrite --core --accessory", shell=True, check=True)
+subprocess.run(python_cmd + " ../poppunk_assign-runner.py --query some_queries.txt --db example_db --model-dir example_refine --output example_query --run-qc --length-range 2900000 3000000 --max-zero-dist 1 --overwrite", shell=True, check=True)
+subprocess.run(python_cmd + " ../poppunk_assign-runner.py --query some_queries.txt --db example_db --model-dir example_refine --output example_query --run-qc --max-pi-dist 0.04 --max-zero-dist 1 --betweenness --overwrite", shell=True, check=True)
+subprocess.run(python_cmd + " ../poppunk_assign-runner.py --query more_queries.txt --db example_db --model-dir example_refine --output example_query --run-qc --max-zero-dist 0.3 --overwrite", shell=True, check=True)
+subprocess.run(python_cmd + " ../poppunk_assign-runner.py --query more_queries.txt --db example_db --model-dir example_refine --output example_query --run-qc --max-zero-dist 1 --max-merge 3 --overwrite", shell=True, check=True)
 subprocess.run(python_cmd + " ../poppunk_assign-runner.py --query some_queries.txt --db example_db --model-dir example_dbscan --output example_query_update --update-db --graph-weights --overwrite", shell=True, check=True) # uses graph weights
 subprocess.run(python_cmd + " ../poppunk_assign-runner.py --query single_query.txt --db example_db --model-dir example_refine --output example_single_query --update-db --overwrite", shell=True, check=True)
+subprocess.run(python_cmd + " ../poppunk_assign-runner.py --query inref_query.txt --db example_db --model-dir example_refine --output example_single_query --write-references", shell=True, check=True) # matched name, but should be renamed in the output
 subprocess.run(python_cmd + " ../poppunk_assign-runner.py --query some_queries.txt --db example_db --model-dir example_refine --model-dir example_lineages --output example_lineage_query --overwrite", shell=True, check=True)
 
+# test updating order is correct
+sys.stderr.write("Running distance matrix order check (--update-db)\n")
+subprocess.run(python_cmd + " test-update.py", shell=True, check=True)
+
 # viz
 sys.stderr.write("Running visualisations (poppunk_visualise)\n")
 subprocess.run(python_cmd + " ../poppunk_visualise-runner.py --ref-db example_db --output example_viz --microreact", shell=True, check=True)
@@ -87,13 +93,9 @@
 subprocess.run(python_cmd + " ../poppunk_visualise-runner.py --ref-db example_db --output example_mst --microreact --tree both", shell=True, check=True)
 subprocess.run(python_cmd + " ../poppunk_mst-runner.py --distance-pkl example_db/example_db.dists.pkl --rank-fit example_lineages/example_lineages_rank5_fit.npz --previous-clustering example_dbscan/example_dbscan_clusters.csv --output example_sparse_mst --no-plot", shell=True, check=True)
 
-# t-sne
-sys.stderr.write("Running tsne viz\n")
-subprocess.run(python_cmd + " ../poppunk_tsne-runner.py --distances example_db/example_db.dists --output example_tsne --perplexity 5 --verbosity 1", shell=True, check=True)
-
-# prune
-sys.stderr.write("Running poppunk_prune\n")
-subprocess.run(python_cmd + " ../poppunk_prune-runner.py --distances example_db/example_db.dists --ref-db example_db --remove subset.txt --output example_prune", shell=True, check=True)
+# mandrake
+sys.stderr.write("Running mandrake viz\n")
+subprocess.run(python_cmd + " ../poppunk_mandrake-runner.py --distances example_db/example_db.dists --output example_mandrake --perplexity 5", shell=True, check=True)
 
 # references
 sys.stderr.write("Running poppunk_references\n")
@@ -108,9 +110,5 @@
 subprocess.run(python_cmd + " ../poppunk-runner.py --citation --fit-model bgmm --ref-db example_db --K 4", shell=True, check=True)
 subprocess.run(python_cmd + " ../poppunk_assign-runner.py --citation --query some_queries.txt --db example_db --output example_query", shell=True, check=True)
 
-# web API
-# sys.stderr.write("Running API tests\n")
-# subprocess.run(python_cmd + " test-web.py", shell=True, check=True)
-
 sys.stderr.write("Tests completed\n")
 
diff --git a/test/test-gpu.py b/test/test-gpu.py
index 9b932cbb..27d08e8a 100755
--- a/test/test-gpu.py
+++ b/test/test-gpu.py
@@ -1,5 +1,5 @@
 #!/usr/bin/env python
-# Copyright 2018-2020 John Lees and Nick Croucher
+# Copyright 2018-2022 John Lees and Nick Croucher
 
 """Tests for PopPUNK using GPUs where possible"""
 
@@ -17,13 +17,9 @@
 else:
     python_cmd = "python"
 
-#easy run
 sys.stderr.write("Running database creation (--create-db)\n")
-subprocess.run(python_cmd + " ../poppunk-runner.py --create-db --r-files references.txt --min-k 13 --k-step 3 --output example_db --qc-filter prune --overwrite --gpu-dist", shell=True, check=True)
-
-# create database with different QC options
-sys.stderr.write("Running database QC test (--create-db)\n")
-subprocess.run(python_cmd + " ../poppunk-runner.py --create-db --r-files references.txt --min-k 13 --k-step 3 --output example_qc --qc-filter continue --length-range 2000000 3000000 --overwrite --gpu-dist", shell=True, check=True)
+subprocess.run(python_cmd + " ../poppunk-runner.py --create-db --r-files references.txt --min-k 13 --k-step 3 --plot-fit 5 --output example_db --overwrite --gpu-dist", shell=True, check=True)
+subprocess.run(python_cmd + " ../poppunk-runner.py --qc-db --ref-db example_db --type-isolate \"12754_4#79\" --overwrite", shell=True, check=True)
 
 # test updating order is correct
 sys.stderr.write("Running distance matrix order check (--update-db)\n")
@@ -83,9 +79,9 @@
 subprocess.run(python_cmd + " ../poppunk_visualise-runner.py --ref-db example_db --output example_mst --microreact --tree both --gpu-graph", shell=True, check=True)
 subprocess.run(python_cmd + " ../poppunk_mst-runner.py --distance-pkl example_db/example_db.dists.pkl --rank-fit example_lineages/example_lineages_rank5_fit.npz --previous-clustering example_dbscan/example_dbscan_clusters.csv --output example_sparse_mst --no-plot --gpu-graph", shell=True, check=True)
 
-# t-sne
-sys.stderr.write("Running tsne viz\n")
-subprocess.run(python_cmd + " ../poppunk_tsne-runner.py --distances example_db/example_db.dists --output example_tsne --perplexity 5 --verbosity 1 --use-gpu", shell=True, check=True)
+# mandrake
+sys.stderr.write("Running mandrake viz\n")
+subprocess.run(python_cmd + " ../poppunk_mandrake-runner.py --distances example_db/example_db.dists --output example_mandrake --perplexity 5 --use-gpu", shell=True, check=True)
 
 # prune
 sys.stderr.write("Running poppunk_prune\n")
diff --git a/test/test-update.py b/test/test-update.py
index 12b1b9d0..73f695b0 100755
--- a/test/test-update.py
+++ b/test/test-update.py
@@ -70,7 +70,7 @@ def old_get_seq_tuples(rows,cols):
 subprocess.run(python_cmd + " ../poppunk-runner.py --fit-model lineage --ref-db batch12 --ranks 1,2", shell=True, check=True)
 subprocess.run(python_cmd + " ../poppunk-runner.py --create-db --r-files rfile1.txt --output batch1 --overwrite", shell=True, check=True)
 subprocess.run(python_cmd + " ../poppunk-runner.py --fit-model lineage --ref-db batch1 --ranks 1,2", shell=True, check=True)
-subprocess.run(python_cmd + " ../poppunk_assign-runner.py --db batch1 --query rfile2.txt --output batch2 --update-db --overwrite", shell=True, check=True)
+subprocess.run(python_cmd + " ../poppunk_assign-runner.py --db batch1 --query rfile2.txt --output batch2 --update-db --overwrite --max-a-dist 1", shell=True, check=True)
 
 # Load updated distances
 X2 = np.load("batch2/batch2.dists.npy")
diff --git a/test/test-web.py b/test/test-web.py
deleted file mode 100644
index 1b2f57ee..00000000
--- a/test/test-web.py
+++ /dev/null
@@ -1,135 +0,0 @@
-import os
-import sys
-import subprocess
-from shutil import copyfile
-
-# testing without install
-#sys.path.insert(0, '..')
-from PopPUNK.assign import assign_query
-from PopPUNK.web import default_options, summarise_clusters, get_colours, api, graphml_to_json
-from PopPUNK.utils import setupDBFuncs
-from PopPUNK.visualise import generate_visualisations
-
-def main():
-    # Copy and move args and sketch files into example dirs
-    copyfile("web_args.txt", "example_db/args.txt")
-    copyfile("example_viz/example_viz_core_NJ.nwk", "example_viz/example_viz.nwk")
-
-    # Test the output of the PopPUNk-web upload route for incorrect data types
-    sys.stderr.write('\nTesting assign for PopPUNK-web\n')
-    with open("json_sketch.txt", "r") as s:
-        sketch = s.read()
-    species = "Listeria monocytogenes"
-    species_db = "example_db"
-    outdir = "example_api"
-    if not os.path.exists(outdir):
-        os.mkdir(outdir)
-    args = default_options(species_db)
-    qc_dict = {'run_qc': False }
-    print("Weights: " + str(args.assign.graph_weights))
-    dbFuncs = setupDBFuncs(args.assign, qc_dict)
-    ClusterResult = assign_query(dbFuncs,
-                                args.assign.ref_db,
-                                args.assign.q_files,
-                                outdir,
-                                qc_dict,
-                                args.assign.update_db,
-                                args.assign.write_references,
-                                args.assign.distances,
-                                args.assign.threads,
-                                args.assign.overwrite,
-                                args.assign.plot_fit,
-                                False, #args.assign.graph_weights,
-                                args.assign.max_a_dist,
-                                args.assign.max_pi_dist,
-                                args.assign.type_isolate,
-                                args.assign.model_dir,
-                                args.assign.strand_preserved,
-                                args.assign.previous_clustering,
-                                args.assign.external_clustering,
-                                args.assign.core_only,
-                                args.assign.accessory_only,
-                                args.assign.gpu_sketch,
-                                args.assign.gpu_dist,
-                                args.assign.gpu_graph,
-                                args.assign.deviceid,
-                                args.assign.web,
-                                sketch,
-                                args.assign.save_partial_query_graph)
-    query, query_prevalence, clusters, prevalences, alias_dict, to_include = \
-        summarise_clusters(outdir, species, species_db)
-    colours = get_colours(query, clusters)
-    url = api(query, "example_viz")
-    sys.stderr.write('PopPUNK-web assign test successful\n')
-    print("Done clustering")
-    # Test generate_visualisations() for PopPUNK-web
-    sys.stderr.write('\nTesting visualisations for PopPUNK-web\n')
-    if len(to_include) < 3:
-        args.visualise.microreact = False
-    generate_visualisations(outdir,
-                            species_db,
-                            os.path.join(outdir, outdir + '.dists'), # distances,
-                            None,
-                            args.visualise.threads,
-                            outdir,
-                            args.visualise.gpu_dist,
-                            args.visualise.deviceid,
-                            args.visualise.external_clustering,
-                            args.visualise.microreact,
-                            args.visualise.phandango,
-                            args.visualise.grapetree,
-                            args.visualise.cytoscape,
-                            args.visualise.perplexity,
-                            args.visualise.strand_preserved,
-                            outdir + "/include.txt",
-                            species_db,
-                            species_db + "/" + os.path.basename(species_db) + "_clusters.csv",
-                            args.visualise.previous_query_clustering,
-                            None, # previous MST
-                            None, # previous distances,
-                            outdir + "/" + os.path.basename(outdir) + "_graph.gt",
-                            args.visualise.gpu_graph,
-                            args.visualise.info_csv,
-                            args.visualise.rapidnj,
-                            args.visualise.tree,
-                            args.visualise.mst_distances,
-                            args.visualise.overwrite,
-                            args.visualise.core_only,
-                            args.visualise.accessory_only,
-                            args.visualise.display_cluster,
-                            web=True)
-    networkJson = graphml_to_json(outdir)
-    if len(to_include) >= 3:
-        with open(os.path.join(outdir, os.path.basename(outdir) + "_core_NJ.nwk"), "r") as p:
-            phylogeny = p.read()
-    else:
-        phylogeny = "A tree cannot be built with fewer than 3 samples."
-
-    # ensure web api outputs are of the correct type
-    if not isinstance(species, str):
-        raise TypeError('"Species" datatype is incorrect, should be string.\n')
-    if not (isinstance(query_prevalence, float) or isinstance(query_prevalence, int)):
-        raise TypeError('"query_prevalence" datatype is incorrect, should be float/integer.\n')
-    if not isinstance(query, str):
-        raise TypeError('"query" datatype is incorrect, should be string.\n')
-    if not isinstance(clusters, list) and not isinstance(clusters[0], str):
-        raise TypeError('"clusters" datatype is incorrect, should be list of strings.\n')
-    if not isinstance(prevalences, list) and not (isinstance(prevalences[0], float) or isinstance(prevalences[0], int)):
-        raise TypeError('"prevalences" datatype is incorrect, should be list of floats/integers.\n')
-    if not isinstance(colours, list) and not isinstance(colours[0], str):
-        raise TypeError('"colours" datatype is incorrect, should be list of strings.\n')
-    if not isinstance(url, str):
-        raise TypeError('"url" datatype is incorrect, should be string.\n')
-    if not isinstance(alias_dict, dict):
-        raise TypeError('"alias_dict" datatype is incorrect, should be dictionary.\n')
-    if not isinstance(outdir, str):
-        raise TypeError('"outdir" datatype is incorrect, should be string.\n')
-    if not isinstance(networkJson, dict):
-        raise TypeError('"networkJson" datatype is incorrect, should be dict.\n')
-    if not isinstance(phylogeny, str):
-        raise TypeError('"phylogeny" datatype is incorrect, should be str.\n')
-
-    sys.stderr.write('\nAPI tests complete\n')
-
-if __name__ == "__main__":
-    main()
diff --git a/test/web_args.txt b/test/web_args.txt
deleted file mode 100644
index dc397c11..00000000
--- a/test/web_args.txt
+++ /dev/null
@@ -1,61 +0,0 @@
-{
-    "assign":{
-        "output":"WebOutput",
-        "update_db":false,
-        "write_references":true,
-        "threads":1,
-        "overwrite":true,
-        "plot_fit":0,
-        "graph_weights":true,
-        "max_a_dist":0.5,
-        "max_pi_dist":0.5,
-        "type_isolate":null,
-        "strand_preserved":false,
-        "external_clustering":null,
-        "core_only":false,
-        "accessory_only":false,
-        "assign_lineage":true,
-        "rank":1,
-        "exact_count":false,
-        "web":true,
-        "gpu_sketch":false,
-        "deviceid":0,
-        "gpu_dist":false,
-        "gpu_graph":false,
-        "min_kmer_count":0,
-        "min_k":14,
-        "max_k":29,
-        "k_step":3,
-        "use_accessory":false,
-        "use_mash":false,
-        "perplexity":20.0,
-        "existing_scheme":null,
-        "rank_list":null,
-        "sketch_sizes":10000,
-        "no_stream":false,
-        "mash":null,
-        "save_partial_query_graph":true
-    },
-    "visualise":{
-        "threads":1,
-        "gpu_dist":false,
-        "gpu_graph":false,
-        "deviceid":0,
-        "external_clustering":null,
-        "microreact":true,
-        "phandango":false,
-        "grapetree":false,
-        "cytoscape":true,
-        "perplexity":20.0,
-        "strand_preserved":false,
-        "previous_query_clustering":null,
-        "info_csv":null,
-        "rapidnj":null,
-        "tree": "nj",
-        "mst_distances":"core",
-        "overwrite":true,
-        "core_only":false,
-        "accessory_only":false,
-        "display_cluster":null
-    }
-}