A Data Scientist’s Guide to Picking an Optimal Approximate Nearest-Neighbor Algorithm
By: Braden Riggs
Tags: Machine Learning, Artificial Intelligence, Computer Science, Benchmarking, Algorithms

Bernhardsson and ANN-Benchmarks to the Rescue:

For this project, we need a little help from the experts. Photo by Tra Nguyen on Unsplash

We have established that there are a range of ANN implementations available for use. However, we need a way of picking out the best of the best, the cream of the crop. This is where Aumüller, Bernhardsson, and Faithfull’s paper ANN-Benchmarks: A Benchmarking Tool for Approximate Nearest Neighbor Algorithms and its corresponding GitHub repository comes to our rescue.

The project, which I have discussed in the past, is a great starting point for choosing the algorithm that is the best fit for your project. The paper uses some clever techniques to evaluate the performance of a number of ANN implementations on a selection of datasets. It has these ANN algorithms solve nearest-neighbor queries to determine the accuracy and efficiency of the algorithm at different parameter combinations. The algorithm uses these queries to locate the 10 nearest data points to the queried point and evaluates how close each point is to the true neighbor, which is a metric called Recall. This is then scaled against how quickly the algorithm was able to accomplish its goal, which it called Queries per Second. This metric provides a great reference for determining which algorithms may be most preferential for you and your project.

Screenshot from an earlier blog where I recreated Bernhardsson’s results benchmarking ANN algorithms on the Gloce-25-angular NLP dataset. Read more here. Image by Author.

Part of conducting this experiment requires picking the algorithms we want to test, and the dataset we want to perform the queries on. Based off of the experiments I have conducted on my previous blogs, narrowing down the selection of algorithms wasn’t difficult. In Bernhardsson’s original project he includes 18 algorithms. Given the performance I had seen in my first blog, using the glove-25 angular natural language dataset, there are 9 algorithms worth considering for our benchmark experiment. This is because some algorithms perform so slowly and so poorly that they aren’t even worth considering in this experiment. The algorithms selected are:

Annoy: Spotify's “Approximate Nearest Neighbors Oh Yeah” ANN implementation.

“Approximate Nearest Neighbors Oh Yeah” ANN implementation. Faiss: The suite of algorithms Facebook uses for large dataset similarity search including Faiss-lsh , Faiss-hnsw , and Faiss-ivf .

The suite of algorithms uses for large dataset similarity search including , , and . Flann: Fast Library for ANN.

Fast Library for ANN. HNSWlib: Hierarchical Navigable Small World graph ANN search library.

Hierarchical Navigable Small World graph ANN search library. NGT-panng: Yahoo Japan’s Neighborhood Graph and Tree for Indexing High-dimensional Data.

Neighborhood Graph and Tree for Indexing High-dimensional Data. Pynndescent: Python implementation of Nearest Neighbor Descent for k-neighbor-graph construction and ANN search.

Python implementation of Nearest Neighbor Descent for k-neighbor-graph construction and ANN search. SW-graph(nmslib): Small world graph ANN search as part of the non-metric space library.

In addition to the algorithms, it was important to pick a dataset that would help distinguish the optimal ANN implementations from the not so optimal ANN implementations. For this task, we chose 1% — or a 10 million vector slice — of the gargantuan Deep-1-billion dataset, a 96 dimension computer vision training dataset. This dataset is large enough for inefficiencies in the algorithms to be accentuated and provide a relevant challenge for each one. Because of the size of the dataset and the limited specification of our hardware, namely the 64GBs of memory, some algorithms were unable to fully run to an accuracy of 100%. To help account for this, and to ensure that background processes on our machine didn’t interfere with our results, each algorithm and all of the parameter combinations were run twice. By doubling the number of benchmarks conducted, we were able to average between the two runs, helping account for any interruptions on our hardware.

This experiment took roughly 11 days to complete but yielded some helpful and insightful results.