Using faiss to search in multidimensional spaces | by Sgjurano | Towards Data Science

In autumn 2019, Avito’s Automated Moderation team launched a fake ads detection service built on top of the the faiss library. It helps us detect whether an image has already been used in another ad, even if it was modified in any way — blurred, cropped, etc.
I would like to talk about the problems that we encountered in the process of implementing this service and the approaches we took to solving them.
In this article I will mainly focus on technical aspects, assuming you know a little about searching in multidimensional spaces.
The first thing I thought about when I was given the task to develop an image retrieval system was its specifications and limitations:
The number of entries. From the very start, we had about 150 million vectors, and now there are more than 240 million.Search time limits. In our case, 300 ms for the 95th percentile.Memory limits. The index had to be small enough to fit ordinary servers, including expected growth for the next two years.Maintainability. Since all our services are implemented on Kubernetes, I really did not want to build a system that needed to be run on physical hardware.Images can be added but not deleted or modified.
The number of entries. From the very start, we had about 150 million vectors, and now there are more than 240 million.
Search time limits. In our case, 300 ms for the 95th percentile.
Memory limits. The index had to be small enough to fit ordinary servers, including expected growth for the next two years.
Maintainability. Since all our services are implemented on Kubernetes, I really did not want to build a system that needed to be run on physical hardware.
Images can be added but not deleted or modified.
Since we need to search among hundreds of millions of vectors quickly enough, exhaustive search is not an option — we need an index that will be in RAM. To fit the vectors there, they need to be compressed.
Once the index is in RAM, to protect it against loss, we will occasionally make backup copies and store these externally. Backups should not require doubling the memory size, even temporarily, since we are talking about dozens of gigabytes.
What remains is to provide for horizontal scaling. The easiest solution in our situation is identical eventually consistent indexes. Then the service instances do not need to know about each other’s existence, and the database will be the only synchronization point.
Our service is written in Python 3.7, PostgreSQL is used to store vectors, and a MinIO has been chosen as backups storage. Faiss plays a key role as an indexing library.
To interact with the outside world, we use the avio http framework, which is our in-house wrapper around aiohttp.
Since asyncio doesn’t work well with fork calls that we need for backups and calling long blocking operations in an asynchronous service is bad practice, all interactions with the index have been moved to a separate process, with which we interact using multiprocessing.Pipe.
For compression, we decided to use Product Quantization. This approach allows us to compress the original vector to 64 bytes.
To speed up the search process, we chose a mixed approach based on Inverted File and HNSW.
As a result, we got an index structure, which in terms of faiss is described by the mysterious line: IVF262144_HNSW32,PQ64. This means that we create an Inverted File for 262144 clusters, the closest of which will be selected using HNSW with 32 neighbors, and all vectors are compressed to 64 bytes using the Product Quantization method.
Here are links to an index selection guideline from the developers of faiss and a benchmark.
What you should pay attention to when looking at the benchmark results:
One query is made to the index to search for 10,000 vectors, and timings are given per one vector.The time is indicated for 16 OpenMP threads. If only one thread is used for search, the time will be about 16 times longer, since parallelism inside faiss library is provided by #pragma omp parallel for.
One query is made to the index to search for 10,000 vectors, and timings are given per one vector.
The time is indicated for 16 OpenMP threads. If only one thread is used for search, the time will be about 16 times longer, since parallelism inside faiss library is provided by #pragma omp parallel for.
It is most appropriate to estimate memory costs in relative terms — in bytes per vector. This way, we can identify memory leaks, should these occur, and easily distinguish them from organic growth due to inserts.
The memory is spent primarily on storing vectors, in the case of IVF262144_HNSW32,PQ64 it is 80 bytes per vector:
64 bytes to store the codes describing the vector.
8 bytes to store the id (stored as int64).
8 bytes to store a pointer to a vector inside the cluster.
The relative memory costs can be calculated with the following formula:
int(faiss.get_mem_usage_kb() * 1024 / index.ntotal)
Some memory can be consumed by the pre-computed distance table, but it is no longer calculated automatically when the estimated size exceeds 2Gb. Its size can be estimated as nlist × pq.M × pq.ksub × float. In our case, 262,144 × 64 × 256 × 4 ≈ 17Gb, where pq.M is the number of Product Quantization components, pq.ksub is 256, since so many clusters can be described with a single byte.
What you should pay attention to when estimating memory costs. The above data is for a static index only: if you add and remove items, memory consumption should be multiplied by at least 2. I suspect that this is due to the dynamic memory allocation to vectors in Inverted File, but I have yet to find support for this hypothesis. If you know the answer, please share it in the comments below.
The convergence of bytes per vector after rolling out the service:
If the problem you are working on does not involve deleting and modifying vectors, you can move most of them to a static index archive and insert new vectors only in a smaller index, merging it with the larger one from time to time, and send search queries to both indexes. So, basically, the classic approach to information retrieval.
Despite using OpenMP inside faiss, any long query sent to large clusters can block the index for a substantial amount of time. To avoid system irregularities, we use ThreadPoolExecutor (faiss is friendly and release GIL).
For faiss to run correctly in a multi-threaded environment, two things need to be done. Firstly, do not allow parallel execution of modifying operations (add, remove) and any other operations. Secondly, limit the number of OpenMP threads during the execution of reading queries in such a way that the total number of threads does not exceed the specified CPU-cores limit. Otherwise, the performance loss is inevitable.
The most convenient way to separate the writing load from the reading load is RWlock, which allows many readers to enter the critical section, but only one writer. For Python, I recommend this package. The number of running OpenMP threads can be adjusted using the faiss.omp_set_num_threads function.
For optimal performance, it makes sense to select the batch size per thread in such a way that query-per-second is maximized. We use a batch size of 5.
It is worth noting that with multithreading, memory consumption increases slightly. It seems like a bug (here is the respective issue).
For more information on multithreading support, check out this faiss wiki article.
New vectors are added to the database, from which these are read every 5 seconds by each instance independently and inserted in the index in batches of 10,000 each (or a smaller batch accumulated since the last update). The insertion speed in the index is sufficiently high for the database to become a bottleneck: about 800 thousand vectors per minute.
One 20-core instance is capable of processing up to 150 rps of searches for 20 nearest neighbors of a single image when we optimizing latency and about 550 qps if we optimizing the througput. Load scaling is very straightforward: just add more instances since they do not know anything about each other.
Since the index is in RAM, its loss is possible if the machine is down. To avoid this problem, dump it regularly to the external storage. We use MinIO for this purpose.
The most convenient way is to make a fork and use Copy-on-Write to get a copy of the index that is almost free in terms of memory consumption and impact on the operation time. Then save this index to disk and upload to the storage. Here, I suggest proceeding from the static index size — in our case, about 80 bytes per vector.
Accordingly, when starting the application, you only need to load the latest copy from the repository and insert the missing data from the database.
We do not use GPU search in the production yet, but I took some measurements to understand the ratio of operation times.
Data: SIFT1M, dimension 128.Query: we are looking for 100 nearest neighbors for 10,000 vectors with different nprobe values.
What you should pay attention to:
The index must fit entirely in the GPU memory. Although there are ways to use multiple GPUs for search, I believe it is not the best approach since such a solution would be harder to maintain.Flat indexes run much faster on GPU, but they are applicable only with a relatively small amount of data (up to tens of millions of vectors at a dimension of 128).When using a PQ64 index, the GPU has an advantage only when polling a very large number of clusters.
The index must fit entirely in the GPU memory. Although there are ways to use multiple GPUs for search, I believe it is not the best approach since such a solution would be harder to maintain.
Flat indexes run much faster on GPU, but they are applicable only with a relatively small amount of data (up to tens of millions of vectors at a dimension of 128).
When using a PQ64 index, the GPU has an advantage only when polling a very large number of clusters.
Faiss is probably the best open-source tool for approximate search today, but like any complex tool, it takes time to get used to.
It’s best to learn it by checking the code now and then, although the faiss documentation is pretty good. You can also send questions to the developers via issues. Usually, they respond fairly quickly.
If you are considering deploying this kind of system on a predetermined set of machines, I suggest taking a look at the vearch system by JD.com. They have done most of the dirty work and made their solution available as open-source, although the documentation is still very rudimentary.