Title: Optimistic Query Routing in Clustering-based Approximate Maximum Inner Product Search
Venue: NeurIPS 2025
Accepted: yes
Keywords: maximum inner product search

clustering-based nearest neighbor search

query routing

sketching

optimism principle
Abstract: Clustering-based nearest neighbor search algorithms partition points into shards to form an index, and search only a subset of shards to process a query. Even though search efficacy is heavily influenced by the algorithm that identifies the shards to probe, it has received little attention in the literature. We study routing in clustering-based maximum inner product search, which includes cosine similarity search. We unpack existing routers and notice the surprising role of optimism. We then take a page from the sequential decision making literature and formalize that insight following the principle of ``optimism in the face of uncertainty.'' In particular, we present a framework that incorporates the moments of the distribution of inner products within each shard to estimate the maximum inner product. We then develop a practical instance of our algorithm that uses only the first two moments to reach the same accuracy as state-of-the-art routers by probing up to $50\%$ fewer points on benchmark datasets without compromising efficiency. Our algorithm is also space-efficient: we design a sketch of the second moment whose size is independent of the number of points and requires $\mathcal{O}(1)$ vectors per shard.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers the k-MIPS problem in the external memory setting.\nThe authors specifically focus on the IVF-based approach and proposed how to rank each cluster (or shard).\nThe representative existing approach computes the inner product of a given query and a cluster center for each cluster, and it accesses the $l$ clusters with the maximum inner products.\nOn the other hand, the proposed approach uses $\\theta_i$ to estimate the maximum inner product obtained from $\\mathcal{P}_i$ by considering its distribution.\nThe first practical solution requires $O(d^{2}|C|)$ space, which can be prohibitive for large $d$ and $|C|$.\nTherefore, the authors proposed to approximate the covariance matrix to reduce the space to $O(t^{2}|C|)$ where $t \\ll d$.\nUnder some assumption, the proposed sketch provides a lower error than Eckhart-Young-Mirsky Theorem.\nExperiments were conducted on real-world datasets, and the results demonstrate that the proposed approach yields a better performance than the baseline approaches.\n\nSTRENGTHS AND WEAKNESSES:\nStrong points\n* The proposed routing is available for all IVF-based MIPS algorithms (generally).\n* The proposed algorithm is designed from theory to practice.\n* Experiments were conducted on real datasets.\n\nWeak points\n* It is well known that IVF-based algorithms are slower than graph-based ones.\n* The setting of Problem 1 is arbitrary.\n* Important experiments are missing.\n\nQUESTIONS:\n- Q1: Why did this work consider an IVF-based algorithm?\nIt is well known that IVF-based algorithms were outperformed by graph-based algorithms (e.g., DiskANN [Subramanya et al., 2019]).\nI agree that IVF-based algorithms are easy to fit into the disk-based setting, but this is not specific to IVF.\nYou can see that DiskANN can deal with MIPS (https://github.com/microsoft/DiskANN), so the authors need to use it as a baseline.\nFrom that paper, DiskANN can obtain high recall (e.g., 95%) within about 5 milliseconds on Deep1B, while OPTIMIST took more on Deep10M.\nThis paper does not claim the merit of IVF compared to the other approaches successfully and thus fails to motivate the IVF work.\n\n- Q2: Section 2.1 uses the $(1+\\delta)/2$ success probability as its objective without justification.\nFrom the application perspective, it is hard to understand why this probability is involved in the problem setting.\nClearly, this setting is derived from the theoretical bounding procedure, and it is not convincing why we need to address Problem 1.\nThis work needs a more convincing setting.\n\n- Q3: In my understanding, the general solution (in Section 2.2) is not available in practice because it uses unknown distribution $\\mathcal{D}$.\nSection 2.3 overcomes this issue by ignoring the data distribution, which is good.\nMy concern here is that, in both cases, the pre-processing needs more computational cost than the original IVF.\nHowever, its (BuildRouterForPartition's) cost has not been analyzed theoretically and empirically.\nBecause eigendecomposition requires a high computational cost (i.e., $O(nd^2 + d^3)$ for $n \\times d$ matrix), the pre-processing would incur a high cost for high-dimensional vectors.\n\n- Q4: The main experimental results show \"recall and \\% points probed.\"\nThe difference of #points probed however does not guarantee to show the difference of running time directly.\nTherefore, the authors need to use time-recall curves.\nIn addition, comparing with state-of-the-art disk-based graph indices is missing.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims to optimize the clustering-based Maximum Inner Product Search (MIPS) and proposed an optimism principle based solution that calculates the scoring function of each cluster more effectively. Finally, experiments are conducted on multiple datasets to test the proposed solution.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**     \n>S1. The paper studies a meaningful problem.  \n>S2. The proposed solution has some new ideas.  \n>S3. Experiments are conducted on multiple datasets.  \n\n**Weaknesses:**  \n>W1. Practical issues are not sufficiently considered in the algorithm design.  \n>W2. Some experimental setups are unclear.  \n>W3. More experiments are suggested.  \n>W4. The presentation requires further improvements.\n\nQUESTIONS:\nQ1. Regarding the algorithm design, please explain how to support dynamic updates and conduct experiments on this practical issue.\n\nQ2. Time complexity of Algorithm 1 should be analyzed.\n\nQ3. Section 2.2 starts with an assumption of Gaussian distribution. To support this assumption, it may be better to present a figure on the data distribution.\n\nQ4. The evaluation mainly assesses recall. However, the major concern seems to be time efficiency. The results would be more convincing if the authors directly report the search time (ie latency as did in the appendix).\n\nQ5. Based on the experimental setup, it seems all compared methods must linearly scan each shard. However, in practice, each shard can be indexed by graph-based MIPS index (eg HNSW [R1]). Since HNSW is highly scalable, the improvement may become marginal when HNSW indexes each shard. Please conduct an additional experiment.  \n[R1] Efficient and Robust Approximate Nearest Neighbor Search Using Hierarchical Navigable Small World Graphs. IEEE TPAMI.\n\nQ6. In both the Introduction and Experimental Setup, the number of clusters is assumed to be $O(\\sqrt{m})$ without references for this setting. When handling many clusters, construction time may be slow. Please report the construction time.\n\nQ7. Some presentation issues:  \n(1) In the references, the venue for \"Unleashing Graph Partitioning for Large-Scale Nearest Neighbor Search\" should be reported.  \n(2) Discussion on the limitations of the proposed solution is missing.  \n(3) Overall, this paper may be a little hard to follow for readers without enough background knowledge.\n\nLIMITATIONS:\nThe paper didn’t point out the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies clustering-based nearest neighbor search. Data structures for this purpose partition the data points into shards, and a query for a point $q$ proceeds in two steps: First, in a routing phase, a set of promising shards is selected. Second, inside each shard, a scoring mechanism is applied to identify the nearest neighbors.\n\nThe submission proposes a novel algorithm for the routing part (the first step above). The algorithm is based on the \"principle of optimism\" from online learning. In a nutshell, it takes an \"optimism parameter\" $\\delta>0$ and assigns thresholds $\\theta_i$ to shards such that with probability at least $(1+\\delta)/2$ all data points in the shard have similarity at most $\\theta_i$ with the query point.\n\nThe new algorithm is based on a one-sided Chebyshev inequality and then it is shown how it can be implemented more efficiently using low-rank approximation of the covariance matrix. This is achieved via SVD and via another, more efficient approach, that employs sketching. This method is motivated through multivariate Gaussians and some additional theoretical guarantees are derived.\n\nIn the experiments, the method is evaluated with data stored on SSDs and on blob storage. It is shown that it outperforms state-of-the-art methods substantially on different real-world datasets.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. Nearest neighbor search is an important and still highly active research topic.\n2. The paper is very well-written and easy to follow.\n3. The new method is well-motivated and easy to implement.\n4. Theoretical guarantees are obtained. While I was initially a bit skeptical about the assumptions of Lemma 2, the appendix contains a short discussion about them. It also provides some properties of real-world datasets, showing the validity of the first assumption.\n\nWeaknesses:\n1. As pointed out in Appendix C.2, it might be possible to remove the second assumption through random rotations. I think this direction could have been explored a bit more (and it also does not seem overly difficult to me).\n\nQUESTIONS:\n1. To remove the second assumption, would a random rotation as used, e.g., in the fast-JL transform work out of the box when applied to the data points?\n2. If the shards were not given as input, is there some hope to compute shards that provide better guarantees for Problem 1?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nSuppose that a large dataset is stored across many shards. The goal, given a query point, is to find the k nearest neighbor (or more specifically, the data points with the top k inner product with the query point). This paper studies the problem of routing, which is the problem of how to summarize the shards and choose which shards to look for the top k data points.\n\nPrior approaches are to summarize each shard by the mean or normalized mean (unit vector) of the data points in the shard, and then search within shards whose mean has maximal inner product with the query point. This paper proposes a simple refinement, where what is stored in not just the mean of shard, but its covariance (or an approximation of it). Using a concentration inequality a la Chebyshev, it is possible to compute an estimate of how large the dot product of the query vector with the top data point in a shard is.\n\nAnother contribution that may be of independent interest was an efficient decomposition for PSD matrices as a sum of a diagonal matrix and a low rank matrix.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\nThe paper studies a fundamental problem and provides a simple heuristic to solve it. It was also written pretty clearly, with nice intuition given throughout. I think this paper raises a problem area that is worth further research, especially motivated by how data is stored and queried in practice.\n\n**Weaknesses**\n\nI think the paper has sufficient material for publication, and worthwhile in that it points out a perhaps understudied problem that arises with extremely large-scale data. But I also think it leaves a lot of fairly obvious questions unanswered, and I think it is also reasonable if the paper were to be published in a future venue with more developed results. Some immediate points that come to mind:\n\n1. Another algorithm this paper may want to compare against is the cover tree [Beygelzimer et. al. 2006], which performs exact nearest neighbor search using O(log m) query time, where m is the number of data points. It seems that the query time for this algorithm achieving approximate nearest neighbor search is on average O(C + m/C), where C is the number of shards. This is optimized to O(sqrt(m)). Of course, there may be real trade-offs between these two approaches, which might mean that the proposed approach is still viable in practice. Perhaps the authors could comment on when they think the proposed approach would have benefits? I can partially answer this question: the cover tree might not admit sharding. Also, the approaches might not be mutually exclusive (the cover tree data structure could be used within the shard, leading to an overall query complexity of O(C + log m / C), or the optimized O((log m)^{1/2})).\n\n2. I think it would be reasonable to also study the case where the algorithm has control over how the shards are generated. For example, is there a large benefit if the shards are guaranteed to have bounded diameter (and so have subgaussian concentration)? Or, is it often the case that the router is constructed independently of the sharding process?\n\n3. White it is fairly clear, it may be nice to have a brief but explicit section on the computational complexity of constructing, updating, and querying this data structure.\n\nQUESTIONS:\nSee above section under weaknesses. I think it may be good to include discussion about the cover tree and related exact and ANN data structures.\n\n[Pre-rebuttal:] Currently, I will recommend a 4: Borderline Accept. I think the problem studied is quite interesting and is worth publishing for a simple solution. But there is a lot left that could be said that would be reasonable to expect this paper to include.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}