SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Example 3 As Fig. 3(c) shows, in a two-dimensional space,  $U(u_{i},u_{0})$  is a perpendicular bisector of the line connecting  $u_{i}$  and  $u_{0}$  (i.e., the red line),  $H(u_{i},u_{0})$  is located on the upper side of  $U(u_{i},u_{0})$ , and  $B(u_{i},\delta (\nu (u_{i}),\nu (u_{0})))$  is the area enclosed by the green circle. Therefore, the landing zone  $L(u_{i},u_{0})$  formed by  $u_{i}$  and  $u_{0}$  is the green shaded region (i.e.,  $H(u_{i},u_{0})\setminus B(u_{i},\delta (\nu (u_{i}),\nu (u_{0})))$ ).
To find the areas without any neighbor of  $u_{i}$ , we intersect the landing zones of all its existing neighbors (i.e.,  $\bigcap_{u_j\in N(u_i)}L(u_i,u_j)$ ). Then, we add the closest vertex (w.r.t.  $u_{i}$ ) within this area into  $N(u_{i})$ , making the neighbors more diverse. We get  $N(u_{i})$  for each vertex  $u_{i}\in V$  as follows (Algorithm 3).
$①$  Candidates acquisition. We obtain a set of  $l$  candidate neighbors for  $u_{i}$  from  $(V\setminus \{u_i\})$  , denoted by  $C(u_{i})$  . We can use random sampling or an extra index to get  $C(u_{i})$  (e.g., we get  $C(u_{i})$  based
on NSW [35] and KGraph [14] in §4.3). We ensure that  $l \geq R$ , where  $R$  is the maximum number neighbors for  $u_i$  (i.e.,  $|N(u_i)| \leq R$ ). ② Neighbors initialization. We sort the elements in  $C(u_i)$  in ascending order of distance to  $u_i$ . We initialize  $N(u_i)$  with  $u_i$ 's nearest candidate neighbor  $u_t$  from  $C(u_i)$  and remove  $u_t$  from  $C(u_i)$ . ③ Neighbors update. We select the nearest vertex  $u_p$  of  $u_i$  from  $C(u_i)$ , and add it to  $N(u_i)$  if it is in the intersection area (i.e.,  $\bigcap_{u_j \in N(u_i)} L(u_i, u_j)$ ) of the landing zones of  $u_i$  and its existing neighbors. We repeat this process until  $C(u_i) = \emptyset$  or  $|N(u_i)| = R$ .
Our strategy ensures that  $u_{i}$ 's neighbors are diverse in terms of the areas where  $C(u_{i})$  is located. This is because we add  $u_{p}$  to  $N(u_{i})$  from a different area each time. We have the following theorem.
Theorem 3 The angle between any two neighbors is not less than  $\pi /3$ . (See Appendix J for the proof)
Example 4 In Fig. 3(c), assuming  $N(u_{i}) = \{u_{0}\}$  and  $C(u_{i}) = \{u_{3}, u_{4}, \dots, u_{9}\}$ , we include  $u_{3}$  in  $N(u_{i})$  because it lies within  $L(u_{i}, u_{0})$  (the green region). Next, we add  $u_{5}$  to  $N(u_{i})$  since it belongs to  $L(u_{i}, u_{0}) \cap L(u_{i}, u_{3})$ . We continue this process until each new area
has one neighbor of  $u_{i}$ . Consequently, in the search procedure, we can efficiently route to the query object  $q$  by utilizing the neighbors of  $u_{i}$  present in the same area as  $q$ .
Algorithm 3: Edge Selection  $(u_i, l, R)$   
Input: Vertex  $u_i$ , constants  $l$  and  $R$   
Output: Neighbor set  $N(u_i)$   
candidate set  $C(u_i) \gets l$  candidate neighbors  
 $u_t \gets \arg \min_{u_t \in C(u_i)} \delta(\nu(u_i), \nu(u_t))$ $N(u_i) \gets N(u_i) \cup \{u_t\}$ ;  
 $C(u_i) \gets C(u_i) \setminus \{u_t\}$   
while  $C(u_i) \neq \emptyset$  or  $N(u_i) < R$  do  
\[
\begin{aligned}
& u_p \gets \arg \min_{u_p \in C(u_i)} \delta(\nu(u_i), \nu(u_p)) \\
& C(u_i) \gets C(u_i) \setminus \{u_p\} \\
& \text{if } u_p \in \bigcap_{u_j \in N(u_i)} L(u_i, u_j) \text{ then} \\
& \quad \left\lfloor N(u_i) \gets N(u_i) \cup \{u_p\} \right\rfloor
\end{aligned}
\]  
return  $N(u_i)$
Complexity. To obtain  $C(u_i)$ , the time complexity depends on the methods used, so we skip this term in our analysis. The time complexity of sorting  $C(u_i)$  is  $O(l \cdot \log(l))$ . For each  $u_p \in C(u_i)$ , we check at most  $|N(u_i)|$  times  $(|N(u_i)| \leq R)$  to see if  $u_p$  is in  $L(u_i, u_j)$  for each  $u_j \in N(u_i)$ . So, our edge selection gets the final  $N(u_i)$  with no more than  $l \cdot R$  checks. Thus, the time complexity of our edge selection on the vertex set  $V$  is  $O(l \cdot (R + \log(l)) \cdot |V|)$ .
4.2 Routing
Routing is a crucial process for searching on a PG. It determines a path from the start vertex to the result vertex that matches the query. The routing strategy affects both the efficiency and accuracy of the search [39]. A recent study [58] splits routing into two stages: the far stage (S1) and the close stage (S2), based on how far they are from the query object.
Our routing. Algorithm 4 illustrates our random two-stage routing strategy. In S1, we randomly select  $\lceil R / h\rceil$  neighbors from  $N(u_{i})$  (where  $1\leq h\leq R$ , and  $R$  is the maximum out-degree), and calculate their distances to the query object  $q$ . This enables us to quickly approach the neighborhood of  $q$ . In S2, we evaluate the distances of all neighbors in  $N(u_{i})$  to  $q$ , following a similar approach to the greedy search [19]. The transition from S1 to S2 occurs when S1 reaches a local optimum, indicated by no further updates to the result set  $R$ . In S2, we continue updating  $R$  by checking all neighbors of the visited vertex. The process terminates when  $R$  no longer receives any updates.
Theorem 4 In the worst case, our routing has the same Recall@k as the current greedy search [19].
Algorithm 4: Routing  $(G,q,P)$    
Input: PG  $G = (V,E)$  query object  $q$  seed set  $P$    
Output: Result set  $R$    
candidate set  $C\gets P$  ; result set  $R\gets P$    
forall  $\{S1,S2\}$  do while  $R$  is updated do  $u_{i}\gets \arg \min_{u_{i}\in C}\delta (\nu (q),\nu (u_{i}))$ $C = C\setminus \{u_i\}$  S1:  $M\leftarrow \lceil R / h\rceil$  random neighbors of  $u_{i}$  S2:  $M\gets$  all neighbors of  $u_{i}$ $C = C\cup M$  forall  $u_{j}\in N(u_{i})$  do  $u_{r}\gets \arg \max_{u_{r}\in R}\delta (\nu (q),\nu (u_{r}))$  if  $\delta (\nu (q),\nu (u_j)) <   \delta (\nu (q),\nu (u_r))$  then  $\lfloor R = R\backslash \{u_r\} ;R = R\cup \{u_j\}$ $C\gets C\cup R;$    
return  $R$
According to Theorem 4 (See Appendix L for the proof), our random strategy in S1 does not reduce the accuracy because most vertices do not require distance calculation from the query in S1 and we can recover a small number of possibly missed vertices in S2.
Complexity. Previous works [36, 19, 58] show that the time complexity of greedy search on a state-of-the-art PG (e.g., HNSW [36], NSG [19]) is  $O(R \cdot \log(|V|))$ , where  $R \ll |V|$  is the maximum number of neighbors per vertex and  $\log(|V|)$  is roughly the average routing path length. In our routing, we use  $l_1$  and  $l_2$  to denote the average routing path lengths in S1 and S2, respectively; then we have  $l_1 + l_2 = \log(|V|)$ . Thus, our routing's time complexity is  $O((\lceil R / h \rceil) \cdot l_1 + R \cdot l_2)$ .
4.3 NPG with Our Edge Selection and Routing
We present two NPGs,  $NPG_{nsw}$  and  $NPG_{kgraph}$ , that use our edge selection and routing strategies on two basal PGs: NSW [35] and KGraph [14]. See Appendix M for implementation details.
5.1 Experimental Setting
Table 1: Statistics of real-world datasets.
Dataset	Dimension	# Base	# Query	LID [33, 18]	Type
UQ-V	256	1,000,000	10,000	7.2	Video + Attributes
Msong	420	992,272	200	9.5	Audio + Attributes
Audio	192	53,387	200	5.6	Audio + Attributes
SIFT1M	128	1,000,000	10,000	9.3	Image + Attributes
GIST1M	960	1,000,000	1,000	18.9	Image + Attributes
Crawl	300	1,989,995	10,000	15.7	Text + Attributes
GloVe	100	1,183,514	10,000	20.0	Text + Attributes
Enron	1,369	94,987	200	11.7	Text + Attributes
Paper	200	2,029,997	10,000	-	Text + Attributes
BIGANN100M	128	100,000,000	10,000	9.3	Image + Attributes
Datasets. We use ten real-world datasets, including one newly released dataset called Paper. They span various modalities, such as video, image, audio, and text. We summarize their main characteristics in Tab. 1. With the exception of the Paper dataset, the remaining datasets solely consist of high-dimensional feature vectors without any original attributes. Therefore, we generate attributes for each object in these datasets using the similar method described in [51, 56]. For instance, in SIFT1M, we augment each image with attributes such as date, location, and size, thereby creating an object set that comprises both feature vectors and attributes.
Compared methods. We compare our HQ methods with seven existing ones that have been used in many high-tech companies. ADBV [54] is a cost-based HQ method proposed by Alibaba. It optimizes IVFPQ [28] for ANNS. Milvus [38, 51] divides the object set through frequently used attributes, and deploys ADBV [54] on each subset. Vearch [29, 32] is developed by Jingdong, which implements the HQ working off Strategy B. NGT [59] is a ANNS library released by Yahoo Japan, which answers a HQ to conduct attribute filtering atop the candidates recalled by NGT (Strategy B). Faiss [17] is a ANNS library developed by Facebook, which answers a HQ based on IVFPQ [28] and Strategy A. SPTAG [37] is a PG-based ANNS library from Microsoft, which answers HQ on Strategy B. Filtered-DiskANN [21] proposes two optimizations based on DiskANN: FilteredVamana and StitchedVamana. FilteredVamana connects vertices with shared attributes. StitchedVamana builds separate graph indexes for each filter and overlays them. NHQ-NPGnsw and NHQ-NPGkgraph are our HQ methods based on NHQ framework integrating two NPGs.
Metrics. We measure the search efficiency by queries per second (QPS), which is the number of queries  $(\# q)$  divided by the search time  $(t)$ , i.e.,  $\# q / t$ . We use the Recall rate to evaluate the search accuracy, which is defined by Eq. 3. Unlike ANNS, hybrid query also requires attribute constraints in Eq. 3, i.e., the elements in  $R \cap T$  must have the same attributes as the query object.
Implementation setup. All codes are written in C++, and are compiled by  $\mathrm{g + + }$  6.5. All experiments are conducted on a Linux server with an Intel(R) Xeon(R) Gold 6248R CPU at  $3.00\mathrm{GHz}$ , and a 755G memory. We use 64 threads to build all the indexes in parallel. We use a single thread for search, which is a common setting in related work [19, 18]. We report the average results from three trials.
Figure 4: HQ performance.
Figure 5: Comparison with new baselines.
5.2 HQ Performance
Fig. 4 shows that our methods outperform existing methods in terms of  $QPS$  vs  $Recall$  trade-off on all datasets. For example,  $\mathrm{NHQ - NPG}_{kgraph}$  achieves two to three orders of magnitude higher  $QPS$  than others when  $Recall@10 > 0.99$ . To execute Filtered-DiskANN, we only test single-attribute queries in Fig. 5. To eliminate other factors, we implement NHQ on DiskANN (NHQ can be easily extended to the current graph index), named NHQ-DiskANN. We keep the same parameters in DiskANN. From the results, NHQ-DiskANN outperforms Filtered-DiskANN, in both memory and disk versions.
(a) BIGANN100M
(b) BIGANN100M
(a) HNSW (SIFT1M)
Figure 6: HQ performance on larger dataset.
Figure 7: Comparison of different strategies.
(b) NSG (SIFT1M)
5.3 Scalability on Larger Dataset
To test the scalability of NHQ, we implement HQ on the state-of-the-art disk-resident PG, DiskANN [27], using "first ANNS, then AF" strategy (AF-DiskANN) and our NHQ (NHQ-DiskANN). Both implementations have the same hyper-parameters for DiskANN. Fig. 6(a) and (b) show the  $QPS$  and Mean I/Os results for different Recall on BIGANN100M dataset. NHQ-DiskANN consistently beats AF-DiskANN by a large margin. For  $QPS$ , NHQ-DiskANN is  $315 \times$  faster than AF-DiskANN when Recall@  $10 = 0.95$ , because NHQ-DiskANN reduces expensive random disk I/Os. For example, to get a recall of 0.95, AF-DiskANN needs 1,303 disk I/Os, while NHQ-DiskANN only needs 30 I/Os, saving about  $97.7\%$  of disk I/Os. This shows the scalability of NHQ to handle larger dataset.
5.4 Ablation Study
Validation of NHQ framework. We test the universality of our NHQ (Strategy C) by applying the HQ model with "decomposition-assembly" to different PGs (including HNSW [36] and NSG [19]) based on "first ANNS, then AF" (Strategy B in Appendix A). We also use IVFPQ [28] in "first AF, then ANNS" (Strategy A in Appendix A) following state-of-the-art implementation [54, 51] because Strategy A does not support PG (L4 in §1.1). We integrate HNSW and NSG into NHQ to form NHQ-HNSW and NHQ-NSG, respectively. As Fig. 7 shows, NHQ outperforms other strategies on different PGs, and keeps stable QPS advantage on different datasets. Due to IVFPQ's limitations, the HQ based on Strategy A have low accuracy, e.g.,  $Recall@10 < 0.8$  on SIFT1M.
(a) SIFT1M
(b) GIST1M
Figure 8: Effect of our edge selection strategy.
(a) SIFT1M
Figure 9: Effect of different PGs.
(b) GloVe
Effect of our edge selection strategy. We compare our edge selection strategy with four existing ones: NGT [59], HNSW [36], HCNNG [39], and NSG [19] on SIFT1M and GIST1M datasets using a recent evaluation framework [52]. All competitors use the same routing inference and distance function. We measure the Speedup-Recall metric, where Speedup is relative to brute force. Fig. 8
shows that our strategy outperforms the others. For example, at Recall@10=0.9 on SIFT1M, our strategy achieves  $1.1 \times$ ,  $1.4 \times$ , and  $25.9 \times$  speedup over HCNNG/NSG, HNSW, and NGT, respectively.
Table 2: Comparison of different routing strategies.
Speedup (vs. HNSW)	Index size (MB)	Memory cost (MB)
HNSW	1.00	790	3,871
HCNNG	1.02	803	3,884
TOGG	1.20	807	3,888
FINGER	1.30	2,150	11,625
Ours	1.21	790	3,871
Effect of our routing strategy. We compare our routing strategy with three existing ones: TOGG [58], FINGER [8], and HCNNG [39] within the HNSW index framework with consistent parameters. Tab. 2 shows the speedup of each optimized strategy over the original HNSW at Recall@10 = 0.9 on SIFT1M. All optimized strategies are faster than the original HNSW, indicating the benefit of optimizing the routing procedure. Our strategy has the lowest storage cost, as it does not require extra structures. Moreover, it delivers a significant improvement in search performance.
Effect of different PGs under NHQ. We compare the performance of three PGs on NHQ: HNSW [36], NSG [19], and our  $NPG_{kgraph}$ . We call them NHQ-HNSW, NHQ-NSG, and NHQ-NPGkgraph respectively. Fig. 9 shows their results on SIFT1M and GloVe. NHQ-NPGkgraph outperforms the others, especially on the harder GloVe dataset.
Memory overhead. NHQ has a higher memory cost than Faisss when considering their best trade-offs. This is a common drawback of graph-based methods compared to PQ-based methods in the ANNS community. This is mainly because graph-based methods build an extra proximity graph index (stored as an adjacency list) to speed up the online search process. While graph-based methods have a higher storage cost, they achieve a significantly better trade-off between accuracy and efficiency and have become the mainstream algorithms in most vector databases (such as Milvus [51]). Our NHQ framework enhances the ability of current graph-based methods to handle ANNS + AF.
(a)  $z = 36$
(b)  $z = 972$
Figure 10: Hybrid query performance of different number of attribute combinations  $(z)$ .
(c)  $z = 2\tilde{6} 244$
Number of attribute combinations. We conduct QPS-Recall comparisons across different attribute dimensions  $\{3,6,9\}$  in Fig. 10. The corresponding number of attribute combinations (represented by  $z$ ) is  $\{36,972,26244\}$ . We find that increasing  $z$  can pose challenges for both the NHQ and existing frameworks. However, NHQ demonstrates higher speedup over existing frameworks in scenarios with a large  $z$ , indicating its potential advantage in handling diverse attribute sets efficiently.
In this paper, we address HQ, which is ANNS with attribute constraint. We present a NHQ framework that adapts to most existing PGs (including disk-resident PGs, e.g., DiskANN [27]) and optimizes them for HQ. We introduce two NPGs with enhanced edge selection and routing strategies that achieve better ANNS performance than original PGs. We integrate several PGs into NHQ to implement HQ methods that are up to  $315 \times$  faster than the state-of-the-art alternatives. A possible future and ongoing work is to apply our framework to multimodal search to handle more complex query requirements.
This work was supported by the Primary R&D Plan of Zhejiang (2021C03156 and 2023C03198) and the National NSF of China (62072149).
[57] Xiaoliang Xu, Jun Liu, Yuxiang Wang, and Xiangyu Ke. Academic expert finding via  $(k,\mathcal{P})$ -core based embedding over heterogeneous graphs. In ICDE, pages 338-351, 2022.
[58] Xiaoliang Xu, Mengzhao Wang, Yuxiang Wang, and Dingcheng Ma. Two-stage routing with optimized guided search and greedy algorithm on proximity graph. Knowledge-Based Systems, page 107305, 2021.
[59] Yahoo. Nearest neighbor search with neighborhood graph and tree for high-dimensional data. https://github.com/yahoojapan/NGT, 2016.
[60] Wen Yang, Tao Li, Gai Fang, and Hong Wei. Pase: Postgresql ultra-high-dimensional approximate nearest neighbor search extension. In SIGMOD, page 2241-2253, 2020.
[61] Zhao Zhang, Fuzhen Zhuang, Hengshu Zhu, Zhiping Shi, Hui Xiong, and Qing He. Relational graph neural network with hierarchical attention for knowledge graph completion. In AAAI, pages 9612-9619, 2020.
[62] Wan-Lei Zhao, Hui Wang, and Chong-Wah Ngo. Approximate k-nn graph construction: A generic online approach. IEEE Transactions on Multimedia, 2021.
[63] Weijie Zhao, Shulong Tan, and Ping Li. SONG: approximate nearest neighbor search on GPU. In ICDE, pages 1033-1044, 2020.
[64] Yuke Zhu, Ce Zhang, Christopher Ré, and Li Fei-Fei. Building a large-scale multimodal knowledge base for visual question answering. CoRR, abs/1507.05670, 2015.