SEARCHING IN ONE BILLION VECTORS: RE-RANK WITH SOURCE CODING
Hervé Jégou
INRIA Rennes

Romain Tavenard
University Rennes I

ABSTRACT
Recent indexing techniques inspired by source coding have beenshown successful to index billions of high-dimensional vectors inmemory. In this paper, we propose an approach that re-ranks theneighbor hypotheses obtained by these compressed-domain indexing methods. In contrast to the usual post-verification scheme, which
performs exact distance calculation on the short-list of hypotheses,
the estimated distances are refined based on short quantization codes,
to avoid reading the full vectors from disk.
We have released a new public dataset of one billion 128dimensional vectors and proposed an experimental setup to evaluate
high dimensional indexing algorithms on a realistic scale. Experiments show that our method accurately and efficiently re-ranks
the neighbor hypotheses using little memory compared to the full
vectors representation.
Index Terms— nearest neighbor search, quantization, source
coding, high dimensional indexing, large databases
1. INTRODUCTION
Approximate nearest neighbors (ANN) search methods [3, 10, 14,15] are required to handle large databases, especially for computer
vision [12] and music retrieval [2] applications. One of the most popular techniques is Euclidean Locality-Sensitive Hashing [3]. However, most of these approaches are memory consuming, since several
hash tables or trees are required. The methods of [4, 15], which embeds the vector into a binary space, better satisfies the memory constraint. They are, however, significantly outperformed in terms ofthe trade-off between memory usage and accuracy by recent methods that cast high dimensional indexing to a source coding problem [11, 5, 6], in particular the product quantization-based methodof [5] exhibits impressive results for large scale image search [6].
State-of-the-art approaches usually perform a re-ranking stage
to produce a ranked list of nearest neighbors. This is always donein partitioning based method such as LSH [3] or FLANN [10], asthe neighbor hypotheses are not ranked on output of the index. But
as shown in [5], this post verification is also important for methods
based on binary [4, 15] or quantized codes [11, 5, 6], as the ranking
provided on output of the large scale search is significantly improved
when verifying the few first hypotheses. In all these approaches, reranking is performed by computing the exact Euclidean distance between each query and the returned hypotheses. For large datasets,
this raises a memory issue: the vectors can not be stored in central
memory and must be read from disk on-the-fly. Moreover, the vectors are to a large extent accessed randomly, which in practice limits
the number of hypotheses that can be verified.
In this paper, we propose a new post-verification scheme in thecontext of compression-based indexing methods. We focus on themethod presented in [5], which offers state-of-the-art performance,
outperforming the FLANN which was previously shown to outperform LSH [10]. It also provides an explicit approximation of theLaurent Amsaleg
CNRS, IRISA

Matthijs Douze
INRIA Grenoble

indexed vectors. Our algorithm exploits the approximation resulting from the first ranking, and refines it using codes stored in RAM.
There is an analogy between this approach and the scalable compression techniques proposed in the last decade [13], where the term
“scalable” means that a reconstruction of the compressed signal isrefined in an incremental manner by successive description layers.
In order to evaluate our approach, we introduce a dataset of one
billion vectors extracted from millions of images using the standard SIFT descriptor [8]. Testing on a large scale is important,as most ANN methods are usually evaluated on sets of unrealistic
size, thereby ignoring memory issues that arise in real applications,where billions of vectors have to be handled [4, 9]. The groundtruth
nearest-neighbors have been computed for 10000 queries using exact distance computations. To our knowledge, this set is the largest
ever released to evaluate ANN algorithms against an exact linear
scan on real data: the largest other experiment we are aware of is the
one reported in [7], where a private set of 179 million vectors wasconsidered. [1] reports an experiment on 1 billion vectors, but on
synthetic data with a known model exploited by the algorithm.
Experiments performed on this dataset show that the proposed
approach offers an alternative to the standard post-verification
scheme. The precision of the search is significantly improvedby the re-ranking step, leading to state-of-the-art performance on
this scale, without accessing the disk.
2. CONTEXT: COMPRESSION BASED INDEXING
In this section, we briefly review the indexing method of [5], whichfinds the approximate k nearest neighbors using a source coding approach. For the sake of presentation, we describe only the Asymmetric Distance Computation (ADC) method proposed in [5]. We
also assume that we search for the nearest neighbor (i.e., k = 1),
Let x ∈ Rd be a query vector and Y = {y1 , . . . , yn } a setof vectors in which we want to find the nearest neighbor NN(x)of x. The ADC approach consists in encoding each vector yi by aquantized version ci = qc (yi ) ∈ Rd . For a quantizer qc (.) with K
centroids, the vector is encoded by bc = log2 (K) bits, assuming K
is a power of 2. An approximate distance dc (x, yi ) between a queryx and a database vector is computed asdc (x, yi )2 = kx − qc (yi )k2 .

(1)

The approximate nearest neighbor NNa (x) of x is obtained byminimizing this distance estimator:
NNa (x) = arg min dc (x, yi )2 = arg min kx − qc (yi )k2 ,
i

i

(2)

which is an approximation of the exact distance calculation
NN(x) = arg min kx − yi k2 .
i

(3)Note that, in contrast with the binary embedding method of [15], thequery x is not converted to a code: there is no approximation error
on the query side.

To get a good vector approximation, K should be large (K =264 for a 64 bit code). For such large values of K, learning a Kmeans codebook is not tractable, neither is the assignment of thevectors to their nearest centroids. To address this issue, [5] uses a
product quantizer, for which there is no need to explicitly enumerate the centroids. A vector y ∈ Rd is first split into m subvectors
y 1 , ..., y m ∈ Rd/m . A product quantizer is then defined as a functionqc (y) = q1 (y 1 ), ..., qm (y m ) ,(4)which maps the input vector y to a tuple of indices by separately
quantizing the subvectors. Each individual quantizer qj (.) has Ks
reproduction values, learned by K-means. To limit the assignment
complexity, O(m × Ks ), Ks is set to a small value (e.g. Ks =256).
However, the set of K centroids induced by the product quantizer
qc (.) is large, as K = (Ks )m . The squared distances in Equation 2
are computed using the decompositionXkxj − qj (y j )k2 ,(5)dc (x, y)2 = kx − qc (y)k2 =
j=1,...,mwhere y j is the j th subvector of y. The squared distances in the sum
are read from look-up tables. These tables are constructed on-the-fly
for a given query, prior to the search in the set of quantization codes,
from each subvector xj and the ks centroids associated with the corresponding quantizer qj . The complexity of the table generation isO(d × Ks ). When Ks  n, this complexity is negligible comparedto the summation cost of O(d × n) in Equation 2.
This approximate nearest neighbor method implicitly sees multidimensional indexing as a vector approximation problem: a database
vector y can be decomposed as
y = qc (y) + r(y),(6)

where qc (y) is the centroid associated with y and r(y) the error vector resulting from the quantization, called the residual vector. It is
proved [5] that the square error between the distance and its estimation is bounded, on average, by the quantization error. This ensures,asymptotically, perfect search results when increasing the number ofbits allocated for the quantization indexes.
The ADC indexing method is fully parametrized by the numberof subvectors m and the total number of bits bc . In the following, we
set bc = 8 (i.e., Ks = 256), as suggested in [5], which means that
we use exactly m bytes per indexed vector.
3. RE-RANKING NEIGHBORS USING SOURCE CODING
3.1. Refinement: principle
The objective of the method proposed in this paper is to avoid thecostly post-verification scheme adopted in most state-of-the-art approximate search techniques [3, 10]. The idea is to take advantage ofthe information on the database point provided by the indexing. Thisis possible when using the ADC method [5]: this search algorithm
provides an explicit approximation qc (y) of database vector y.
We first assume that the first retrieval stage returns a set of k0
hypotheses. These vectors are the one for which a post-verificationis required. For each database vectors yi , the error vector is equal tor(y) = y − qc (y).

(7)

The proposed method consists in reducing the energy of thisresidual vector to limit the impact of the approximation error on the

drdc

Fig. 1. Illustration of the proposed refinement process. For eachdatabase vector y, the distance dc (x, y) = d(x, qc (y)) is computed
to build the short-list of potential nearest neighbors. For selected yvectors, the distance is re-estimated by dr (x, y), which is obtainedby computing the distance between y and its improved approximation dr = qc (y) + qr (y − qc (y)).
estimated distances. This is done by encoding the residual vectorr(y) using another product quantizer qr defined by its reproduction
values Cr :qr (r(y)) = arg min kr(y) − ck2 ,
(8)
c∈Cr

where the product quantizer qr (.) is learned on an independent set ofresidual vectors. Similar to qc (.), the set of reproduction values Cr isnever exhaustively listed, as all operations are performed using the
product space decomposition.
The coded residual vector can be seen as the “least significant
bits”, except that the term “bits” usually refers to scalar quantization. An improved estimation ŷ of y is the sum of the approximation
vector and the decoded residual vector:ŷ = qc (y) + qr (r(y)).

(9)

As shown in Figure 1, this estimator will be used at searchtime to update the distance estimation between the query x and the
database vectors y that are selected as potential neighbors:d(x, y)2 ≈ dr (x, y)2 = kqc (y) + qr (r(y)) − xk2 .

(10)

The refinement product quantizer qr is parametrized by its number of subquantizers and the total number of bits. Similar to qc , weuse 8 bits per subquantizer. Therefore the only parameter for therefinement quantizer is the number m0 of bytes for each code. The
total memory usage per indexed vector is m + m0 bytes.
3.2. Algorithm
This subsection details how the refinement codes are used to re-rank
the hypotheses provided by the ADC. The resulting approach will
be denoted by ADC+R in the experimental section. As for mostindexing algorithms, we distinguish between the offline stage andthe query stage, during which the system needs to be very efficient.
The offline stage consists in learning the indexing parameters andbuilding the indexing structure associated with a vector dataset. It is
performed as follows.
1. The quantizers qc (.) and qr (.) are learned on a training set.
2. The vector dataset Y = {y1 , . . . , yn } to be indexed is encoded using qc , producing codes qc (yi ) for i = 1, . . . , n.

3. The residual vectors are encoded, producing the codesqr (yi − qc (yi )) associated with all the indexed vectors.
Searching a query vector x proceeds as follows:
1. The ADC distance estimation is used to generate a list L of
k0 hypotheses. The selected vectors minimize the estimatorof Equation 5, which is computed directly in the compressed
domain [5].
2. For each vector yi ∈ L, the approximate vector ŷi is explicitly reconstructed using the first approximation qc (yi ) and thecoded residual vector qr (yi ), see Equation 9. The squared
distance estimator d(x, ŷi )2 is subsequently computed.

MethodADC
ADC+R
IVFADCIVFADC+R

m0
0
8
16
32
0
8
16
32

recall@1
0.075
0.258
0.434
0.656
0.088
0.262
0.429
0.630

@10
0.274
0.683
0.895
0.970
0.372
0.701
0.894
0.977

@100
0.586
0.951
0.982
0.985
0.733
0.962
0.982
0.983

time/query
5.626
5.686
5.692
5.689
0.074
0.116
0.119
0.120

Table 1. Performance and efficiency measured on 1 billion vectors,
m=8. The query time is measured in seconds per query. The timings
validate the limited impact of the re-ranking stage on efficiency.

3. The vectors of L associated with the k smallest refined distances are computed.
On output, we obtain a re-ranked list of k approximate nearest
neighbors. The choice of the number k0 of vectors in the short-list
depends on parameters m, m0 , k0 and on the distribution of the vectors. In order for the post-verification scheme to have a negligible
complexity, we typically set the ratio k0 /k to 2.

vector. The groundtruth for smaller sets (n=1M, 2M, 5M, 10M, ...,
200M vectors) is also provided. As our own approach does not require many training vectors, we only used the first million vectors
from the learning set. All measurements (accuracy and timings)
were averaged over the 1000 first queries.

3.3. Non exhaustive variant

4.2. Evaluation protocol

Up to now, we have only considered the ADC method of [5], which
requires an exhaustive scan of the dataset codes. Note however thatthe re-ranking method proposed in this paper can be applied to any
method for which an approximate reconstruction of the indexed vectors is possible, e.g., [11]. In particular, in the experimental section we evaluate our approach with the IVFADC variant of [5], thatavoids the aforementioned exhaustive scan by using an inverted filestructure. This requires an additional coarse quantizer.
Adapting our re-ranking method to the IVFADC method is
straightforward, as this method also provides an explicit approximation of the indexed vectors. In addition to the numbers m and m0of bytes used to encode the vector, this variant requires two additional parameters: the number c of reproduction values of the coarse
quantizer and the number v of inverted lists that are visited for agiven query. The main advantage of this variant is to compute thedistance estimators only for a fraction (in the order of v/c) of thedatabase, at the risk of missing some nearest neighbors if v/c is notlarge enough. Note finally that the memory usage is increased bylog2 (c) bits (typically 4 bytes), due to the inverted file structure. In
the following, the IVFADC method used jointly with our re-ranking
method will be denoted by IVFADC+R.

The search quality is measured by the recall@r measure, i.e., theproportion of queries whose nearest neighbor is ranked in the firstr positions. The curve obtained by varying r corresponds to the
distribution function of the ranks, and the point r=1 corresponds1
to the “precision” measure used in [10] to evaluate ANN methods.
Also, the recall@r is the fraction of queries for which the nearest
neighbor would be retrieved correctly if a short-list of k = r vectors
was verified using exact Euclidean distances.
The efficiency is measured by actual timings.

4. EXPERIMENTS
4.1. BIGANN: a billion-sized evaluation dataset
To evaluate ANN search methods, we propose a new evaluation
dataset available online: http://corpus-texmex.irisa.fr.
This benchmark, called BIGANN, consists of 128-dimensional SIFT
descriptors (widely adopted image descriptors [8]) extracted fromapproximately 1 million images. It comprises three distinct subsets:• base vectors: one billion vectors to search in

4.3. Evaluation of the proposed approach
Unless explicitly specified, our approach is evaluated by querying in
the whole BIGANN set (i.e., one billion vectors). The performanceof the ADC and IVFADC algorithms are given for reference, andcompared with the re-ranked versions ADC+R and IVFADC+R. In
all experiments, we have set k=10000 and k0 =20000. In addition,
for the IVFADC+R we have fixed c=8192 and v=64, which meansthat the query is compared to approximately 1/128th of the indexed
vectors.
The re-ranking gain: We first consider the improvement brought by
the re-ranking stage compared with the reference methods. Figure 2
shows the importance of this re-ranking stage on the recall@r measure: the performance of PQ+R (resp. IVFPQ+R) is significantly
better than that of ADC (resp. IVFADC).
These observations are confirmed by Table 1, which additionallyshows that the impact of the re-ranking stage on the query time islimited. As already reported in [5], the IVFADC version is better
than the ADC method, at the cost of 4 additional bytes per indexed
vector. As expected, this observation remains valid when comparing
IVFADC+R with ADC+R.

• learning vectors: a set of 100 million vectors to compute theparameters involved in the indexing method.

Performance for a fixed memory usage: Table 2 shows that, fora given memory usage (i.e., for m + m0 constant), the re-ranking
approach ADC+R achieves similar recall as ADC, at a lower computing cost. The search is approximately two times faster, as the

The groundtruth has been pre-computed: for each query, we provide the k nearest neighbors that are obtained when computing exact Euclidean distance, as well as their square distance to the query

1 In practice, we are often interested in retrieving the k nearest neighbors(k > 1) and not only the nearest neighbor. We do not include these measuresin the paper, as qualitative conclusions for k=1 remain valid for k > 1.

• query vectors: 10000 vectors that are submitted to the system

1

0.8

0.8

Recall@10

1

Recall

0.6

0.4
ADC
ADC+R, m’=8
ADC+R, m’=32
IVFADCIVFADC+R, m’=8
IVFADC+R, m’=32

0.2

10

100

1000

r

Fig. 2. Searching in one billion vectors: impact of the re-ranking
stage on the search accuracy (recall@r). m=8.
bytes/vector
8
16
32
64

m
8
4
16
8
32
16
64
32

m0
0
4
0
8
0
16
0
32

recall@1
0.075
0.056
0.245
0.258
0.487
0.571
0.791
0.832

@10
0.274
0.223
0.671
0.683
0.956
0.977
1.000
0.999

0.4

0.2

0
1

0.6

@100
0.586
0.504
0.952
0.951
0.999
1.000
1.000
1.000

Table 2. Comparative accuracy of ADC (m0 =0) and ADC+R for
the same total amount of memory (bytes per indexed vector). In
these experiments, the ADC+R approach with m = m0 is approximately 2× more efficient that the ADC approach, as the cost of thethe re-ranking stage is almost negligible, as shown Table 1. Theseexperiments have been performed on the whole BIGANN set.
search time is dominated by the first retrieval stage, whose complexity is asymptotically linear in m and n for large values of n. One canalso observe that near perfect neighbors are obtained by increasing
the number m0 of bytes used to re-rank the queries.
Impact of the dataset size: Figure 3 shows the impact of thedatabase size on the recall@10 measure. The re-ranking stage becomes more important as the database grows in size, due to anincreasing number of outliers. Interestingly, the quality of the searchdegrades more gracefully when using a precise post-verification
scheme (with m0 =16 bytes).
5. CONCLUSION
In this paper, following recent works on multi-dimensional indexing
based on source coding, we propose a method to re-rank the vectors
with a limited amount of memory, thereby avoiding costly disk accesses. Refining the neighbor hypotheses strongly improves recall at
a rather low cost on response time. Our experimental validation is
performed on a new public dataset of one billion vectors.
6. ACKNOWLEDGEMENTS
This work was partly realized as part of the Quaero Programme,
funded by OSEO, French State agency for innovation.

0
1M

ADC
ADC+R, m’=8
ADC+R, m’=16
IVFADCIVFADC+R, m’=8
IVFADC+R, m’=16
10M
100M
Size of the database: number of indexed vectors

1G

Fig. 3. Impact of the database size on search performance measured
by recall@10. m=8.
7. REFERENCES[1] A. Camerra, T. Palpanas, J. Shieh, and E. Keogh. iSAX 2.0:
Indexing and mining one billion time series. In ICDM, September 2010.
[2] M. Casey, R. Veltkamp, M. Goto, M. Leman, C. Rhodes, and
M. Slaney. Content-based music information retrieval: Current directions and future challenges. Proceedings of the IEEE,
96(4):668–696, April 2008.
[3] M. Datar, N. Immorlica, P. Indyk, and V. Mirrokni. Localitysensitive hashing scheme based on p-stable distributions. In
Proceedings of the Symposium on Computational Geometry,pages 253–262, 2004.
[4] H. Jégou, M. Douze, and C. Schmid. Improving bag-offeatures for large scale image search. IJCV, 87(3):316–336,
February 2010.
[5] H. Jégou, M. Douze, and C. Schmid. Product quantization for
nearest neighbor search. PAMI, 33(1):117–128, January 2011.
[6] H. Jégou, M. Douze, C. Schmid, and P. Pérez. Aggregating local descriptors into a compact image representation. In CVPR,
June 2010.
[7] H. Lejsek, F. H. Asmundsson, B. P. Jónsson, and L. Amsaleg.
Nv-tree: An efficient disk-based index for approximate searchin very large high-dimensional collections. PAMI, 31(5):869–
883, May 2009.
[8] D. Lowe. Distinctive image features from scale-invariant keypoints. IJCV, 60(2):91–110, 2004.
[9] A. Mikulik, M. Perdoch, O. Chum, and J. Matas. Learning afine vocabulary. In ECCV, September 2010.
[10] M. Muja and D. G. Lowe. Fast approximate nearest neighborswith automatic algorithm configuration. In VISAPP, 2009.
[11] H. Sandhawalia and H. Jégou. Searching with expectations. In
ICASSP, Signal Processing. IEEE, March 2010.
[12] G. Shakhnarovich, T. Darrell, and P. Indyk. Nearest-Neighbor
Methods in Learning and Vision: Theory and Practice, chapter 3. MIT Press, March 2006.
[13] D. Taubman. High performance scalable image compression with EBCOT. IEEE Transactions on Image Processing,7(9):1158–1170, 2000.
[14] A. Torralba, R. Fergus, and Y. Weiss. Small codes and large
databases for recognition. In CVPR, June 2008.
[15] Y. Weiss, A. Torralba, and R. Fergus. Spectral hashing. In
NIPS, November 2008.