A Greedy Approach for Budgeted Maximum Inner Product Search

Maximum Inner Product Search (MIPS) is an important task in many machine learning applications such as the prediction phase of low-rank matrix factorization models and deep learning models. Recently, there has been substantial research on how to perform MIPS in sub-linear time, but most of the existing work does not have the flexibility to control the trade-off between search efficiency and search quality. In this paper, we study the important problem of MIPS with a computational budget. By carefully studying the problem structure of MIPS, we develop a novel Greedy-MIPS algorithm, which can handle budgeted MIPS by design. While simple and intuitive, Greedy-MIPS yields surprisingly superior performance compared to state-of-the-art approaches. As a specific example, on a candidate set containing half a million vectors of dimension 200, Greedy-MIPS runs 200x faster than the naive approach while yielding search results with the top-5 precision greater than 75%. Introduction In this paper, we study the computational issue in the prediction phase for many embedding based models such as matrix factorization and deep learning models in recommender systems, which can be mathematically formulated as a Maximum Inner Product Search (MIPS) problem. Specifically, given a large collection of n candidate vectors: H = { hj ∈ R : 1, . , n } and a query vector w ∈ R, MIPS aims to identify a subset of candidates that have top largest inner product values with w. We also denote H = [h1, . ,hj , . ,hn] as the candidate matrix. A naive linear search procedure to solve MIPS for a given query w requires O(nk) operations to compute n inner products and O(n log n) operations to obtain the sorted ordering of the n candidates. Recently, MIPS has drawn a lot of attention in the machine learning community due to its wide applicability, such as the prediction phase of embedding based recommender systems [6, 7, 10]. In such an embedding based recommender system, each user i is associated with a vector wi of dimension k, while each item j is associated with a vector hj of dimension k. The interaction (such as preference) between a user and an item is modeled by w i hj . It is clear that identifying top-ranked items in such a system for a user is exactly a MIPS problem. Because both the number of users (the number of queries) and the number of items (size of vector pool in MIPS) can easily grow to millions, a naive linear search is extremely expensive; for example, to compute the preference for all m users over n items with latent embeddings of dimension k in a recommender system requires at least O(mnk) operations. When both m and n are large, the prediction procedure is extremely time consuming; it is even slower than the training procedure used to obtain the m+n embeddings, which ∗Work done while at the University of Texas at Austin. 31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA. costs only O(|Ω|k) operations per iteration, where |Ω| is number of observations and is much smaller than mn. Taking the yahoo-music dataset as an example, m = 1M , n = 0.6M , |Ω| = 250M , and mn = 600B 250M = |Ω|. As a result, the development of efficient algorithms for MIPS is needed in large-scale recommender systems. In addition, MIPS can be found in many other machine learning applications, such as the prediction for a multi-class or multi-label classifier [16, 17], an object detector, a structure SVM predicator, or as a black-box routine to improve the efficiency of learning and inference algorithm [11]. Also, the prediction phase of neural network could also benefit from a faster MIPS algorithm: the last layer of NN is often a dense fully-connected layer, so finding the label with maximum score becomes a MIPS problem with dense vectors [6]. There is a recent line of research on accelerating MIPS for large n, such as [2, 3, 9, 12–14]. However, most of them do not have the flexibility to control the trade-off between search efficiency and search quality in the prediction phase. In this paper, we consider the budgeted MIPS problem, which is a generalized version of the standard MIPS with a computation budget: how to generate a set of top-ranked candidates under a given budget on the number of inner products one can perform. By carefully studying the problem structure of MIPS, we develop a novel Greedy-MIPS algorithm, which handles budgeted MIPS by design. While simple and intuitive, Greedy-MIPS yields surprisingly superior performance compared to existing approaches. Our Contributions: • We develop Greedy-MIPS, which is a novel algorithm without any nearest neighbor search reduction that is essential in many state-of-the-art approaches [2, 12, 14]. • We establish a sublinear time theoretical guarantee for Greedy-MIPS under certain assumptions. • Greedy-MIPS is orders of magnitudes faster than many state-of-the-art MIPS approaches to obtain a desired search performance. As a specific example, on the yahoo-music data sets with n = 624, 961 and k = 200, Greedy-MIPS runs 200x faster than the naive approach and yields search results with the top-5 precision more than 75%, while the search performance of other state-of-the-art approaches under the similar speedup drops to less than 3% precision. • Greedy-MIPS supports MIPS with a budget, which brings the ability to control of the trade-off between computation efficiency and search quality in the prediction phase. Existing Approaches for Fast MIPS Because of its wide applicability, several algorithms have been proposed for efficient MIPS. Most of existing approaches consider to reduce the MIPS problem to the nearest neighbor search problem (NNS), where the goal is to identify the nearest candidates of the given query, and apply an existing efficient NNS algorithm to solve the reduced problem. [2] is the first MIPS work which adopts such a MIPS-to-NNS reduction. Variants MIPS-to-NNS reduction are also proposed in [14, 15]. Experimental results in [2] show the superiority of the NNS reduction over the traditional branchand-bound search approaches for MIPS [9, 13]. After the reduction, there are many choices to solve the transformed NNS problem, such as locality sensitive hashing scheme (LSH-MIPS) considered in [12, 14, 15], PCA-tree based approaches (PCA-MIPS) in [2], or K-Means approaches in [1]. Fast MIPS approaches with sampling schemes have become popular recently. Various sampling schemes have been proposed to handle MIPS problem with different constraints. The idea of the sampling-based MIPS approach is first proposed in [5] as an approach to perform approximate matrix-matrix multiplications. Its applicability on MIPS problems is studied very recently [3]. The idea behind a sampling-based approach called Sample-MIPS, is about to design an efficient sampling procedure such that the j-th candidate is selected with probability p(j): p(j) ∼ hj w. In particular, Sample-MIPS is an efficient scheme to sample (j, t) ∈ [n] × [k] with the probability p(j, t): p(j, t) ∼ hjtwt. Each time a pair (j, t) is sampled, we increase the count for the j-th item by one. By the end of the sampling process, the spectrum of the counts forms an estimation of n inner product values. Due to the nature of the sampling approach, it can only handle the situation where all the candidate vectors and query vectors are nonnegative. Diamond-MSIPS, a diamond sampling scheme proposed in [3], is an extension of Sample-MIPS to handle the maximum squared inner product search problem (MSIPS) where the goal is to identify candidate vectors with largest values of (hj w) . However, the solutions to MSIPS can be very different from the solutions to MIPS in general. For example, if all the inner product values are negative, the ordering for MSIPS is the exactly reverse ordering induced by MIPS. Here we can see that the applicability of both Sample-MIPS and Diamond-MSIPS to MIPS is very limited.