The Impacts of Data, Ordering, and Intrinsic Dimensionality on Recall in Hierarchical Navigable Small Worlds

Venue: ACM_org_SIGIR_ICTIR_2024_Conference
Published: 2024-04-25 09:59:14

Abstract: Vector search systems, pivotal in AI applications, often rely on the Hierarchical Navigable Small Worlds (HNSW) algorithm. However, the behaviour of HNSW under real-world scenarios using vectors generated with deep learning models remains under-explored. Existing Approximate Nearest Neighbours (ANN) benchmarks and research typically has an over-reliance on simplistic datasets like MNIST or SIFT1M and fail to reflect the complexity of current use-cases. Our investigation focuses on HNSW's efficacy across a spectrum of datasets, including synthetic vectors tailored to mimic specific intrinsic dimensionalities, widely-used retrieval benchmarks with popular embedding models, and proprietary e-commerce image data with CLIP models. We survey the most popular HNSW vector databases and collate their default parameters to provide a realistic fixed parameterisation for the duration of the paper.

We discover that the recall of approximate HNSW search, in comparison to exact K Nearest Neighbours (KNN) search, is linked to the vector space's intrinsic dimensionality and significantly influenced by the data insertion sequence. Our methodology highlights how insertion order, informed by measurable properties such as the pointwise Local Intrinsic Dimensionality (LID) or known categories, can shift recall by up to 12 percentage points. We also observe that running popular benchmark datasets with HNSW instead of KNN can shift  rankings by up to three positions for some models. This work underscores the need for more nuanced benchmarks and design considerations in developing robust vector search systems using approximate vector search algorithms. This study presents a number of scenarios with varying real world applicability which aim to better increase understanding and future development of ANN algorithms and embedding models alike.

Main paper: 
 introduction: 
 The efficient retrieval of nearest neighbours in high-dimensional spaces is a requirement for many Artificial Intelligence (AI) applications. This need has driven the development and widespread adoption of Approximate Nearest Neighbours (ANN) algorithms, among which the Hierarchical Navigable Small Worlds (HNSW) [28] algorithm has emerged as a preeminent choice for search and recommendation applications.Despite its extensive utilization, the behaviour and performance of the HNSW algorithm under real-world conditions remains insufficiently explored. This gap in understanding is particularly critical given the evolving nature of datasets in contemporary AI applications. Existing benchmarks for evaluating ANN systems, which often rely upon simplistic or lower-dimensional datasets (MNIST [13] , SIFT1M [21] , etc.), do not adequately reflect many popular real-world use-cases. These datasets are highly curated and do not contain vectors from machine learning embedding models. This discrepancy raises questions about the applicability and reliability of these benchmarks in guiding the design and implementation of vector search systems in real-world scenarios.To bridge the gap between benchmarks and contemporary applications, our research studies the behaviour of HNSW search across vector spaces produced with various methods including synthetic data, popular retrieval benchmarks with popular text embedding models, and real-world e-commerce data with multimodal embeddings from CLIP [35] models.We collate a survery of popular HNSW vector search systems and their default parameters to provide a fixed parameterisation of the algorithm, unlike prior research we study the behaviour of HNSW as a function of data, models, and indexing conditions rather than of parameterisation. This methodology allows us to identify a relationship between both the intrinsic dimensionality of vector spaces as a whole as well as the Local Intrinsic Dimensionality (LID) of vectors within the dataset and the order in which they are added. By controling insertion order of data with LID we observe recall dropping by as much as 12.8 percentage points. We extend this observation about ordering to real world applications and quantify the difference in intrinsic dimensionality between a dataset and its constituent categories. By indexing data with different orders of categories we are able to vary recall by up to 8 percentage points.Through this work, we aim not only to contribute valuable insights into the HNSW algorithm's behaviour, but also to challenge the prevailing benchmarks for ANN search, encouraging the development of more robust and reliable vector search systems.In this work we provide a survey of existing vector search systems that use HNSW and the parameters that they provide as default. Many of these default parameterisations are not documented or easily accessible and are only mentioned in code. We provide the default parameters at time of writing in this work (Table 1 ).In addition to the findings in this work we also release a large 500GB collection of all the vectors used in this research 1 . This includes the complete embeddings from seventeen popular opensource models on the 7 datasets used in this work as well as MS-MARCO [5] . To compliment these embeddings we provide the pointwise Local Intrinsic Dimensionality (LID) estimates for every vector with its 100 nearest neighbours (which is a time consuming 𝑂 (𝑛2 ) task). 
related work: 
 The impacts of intrinsic dimensionality, specifically LID, have been studied by Aumüller et al [4] . In their work it was identified that query sets of varying difficulty would be constructed by identifying the LID of the queries, the impact of this being that averaging results across all queries could mask behaviour of the algorithms in benchmarking.In the source code for the implementation of the original paper, HNSWLib. The authors make reference to a relationship between the 𝑀 parameter and the intrinsic dimensionality of the data, stating that a higher 𝑀 of 48-64 is required for good recall on data with higher intrinsic dimensionality. However, this relationship is not considered in detail in the original work and does not appear to be quantified in research [28] .In other work, P. Lin et al analyse the search time behaviour of HNSW on data with varying LIDs and identify that the hierarchical component of HNSW offers less benefit over a flat search as the LID of the data increases. If the graph contains close neighbourhoods with minimal intersection between their nearest neighbour lists, the search can find it difficult to jump from one local minima to another. This results in worse recall for given parameters, or worse latency for a given recall, as the LID of the data increases [27] .The "curse of dimensionality" is widely acknowledged as a feature which impacts the efficacy of retrieval systems and measures of similarity in general. However, data with a large apparent dimensionality can often have a low intrinsic dimensionality. For some KNN algorithms such as KD-Trees, a sufficiently small intrinsic dimensionality can reduce the likelihood of a low quality solution. For other KNN algorithms a lower intrinsic dimensionality can be indicative of potentially favourable performance with dimensionality reduction techniques applied. [8] . 
the hnsw algorithm and its implementations: 
 For this research we focus on more pure implementations of the HNSW algorithm to avoid introducing additional variables into the experiments. The HNSW implementations from FAISS [14, 22] and HNSWLib [28] are used in this paper as they create a single HNSW graph for all data and do not have any complexities around sharding and segmentation for horizontal scaling and mutability 2 .The HNSW algorithm has three primary parameters that impact recall, memory, and latency: 𝑀, 𝑒 𝑓 𝐶𝑜𝑛𝑠𝑡𝑟𝑢𝑐𝑡𝑖𝑜𝑛, and 𝑒 𝑓 𝑆𝑒𝑎𝑟𝑐ℎ. 𝑀 is the number of bidirectional links to form between each node in the graph, the final layer of the graph typically uses 2 • 𝑀 links; this impacts the recall, memory usage, and latency where higher 𝑀 gives better quality retrieval but worse performance. 𝑒 𝑓 𝐶𝑜𝑛𝑠𝑡𝑟𝑢𝑐𝑡𝑖𝑜𝑛 is the number of candidates to hold in the heap when constructing the graph, evaluating more candidates gives better graphs with higher recall, however it does increase the time spent indexing; 𝑒 𝑓 𝐶𝑜𝑛𝑠𝑡𝑟𝑢𝑐𝑡𝑖𝑜𝑛 does not impact search latency. 𝑒 𝑓 𝑆𝑒𝑎𝑟𝑐ℎ is the size of the candidate list to hold in the heap at search time, higher 𝑒 𝑓 𝑆𝑒𝑎𝑟𝑐ℎ can increase recall at the cost of latency.Methodology for Determining Parameters and Reasoning. We fix the HNSW graph parameters for all experimentation. We acknowledge that many challenges regarding recall for approximate nearest neighbours with HNSW can be circumnavigated by increasing 𝑀, 𝑒 𝑓 𝐶𝑜𝑛𝑠𝑡𝑟𝑢𝑐𝑡𝑖𝑜𝑛, and/or 𝑒 𝑓 𝑆𝑒𝑎𝑟𝑐ℎ. However, in reality it is not feasible to extensively search the parameter space for optimal parameters, and furthermore, it is not feasible to scale these parameters beyond a point as latency degrades.To determine appropriate fixed defaults for this experimentation, we surveyed approximate nearest neighbours systems that use HNSW to determine their defaults. MarqoV1 [32] 16 128 𝑘 HNSWLib [16] 16 200 10 FAISS [38] 32 40 16 Chroma [10] 16 100 10 Weaviate [49] 64 128 100 Qdrant [34] 16 100 128 Milvus [29] 18 240 No Default Vespa [42] 16 200 𝑘 Opensearch (nmslib) [32] 16 512 512 Opensearch (Lucene) [32] 16 512 𝑘 Elasticsearch (Lucene) [15] 16 100No Default Redis [37] 16 200 10 PGVector [33] 16 64 40 Note: In this table, 𝑘 represents the number of results to return. Systems with 𝑒 𝑓 𝑆𝑒𝑎𝑟𝑐ℎ = 𝑘 do not specify a default 𝑒 𝑓 𝑆𝑒𝑎𝑟𝑐ℎ and set it to 𝑘 at seach time.It is clear that many HNSW implementations rely upon relatively low defaults for the algorithm parameters 3 . It is clear that 𝑀 = 16 is widely accepted as a sensible default parameter for the number of bidirectional links to form in the graph, 16 is the median. Values for 𝑒 𝑓 𝐶𝑜𝑛𝑠𝑡𝑟𝑢𝑐𝑡𝑖𝑜𝑛 vary more widely ranging from 40 to 512, for the work we opt to fix it at 𝑒 𝑓 𝐶𝑜𝑛𝑠𝑡𝑟𝑢𝑐𝑡𝑖𝑜𝑛 = 128 as this is the median. 𝑒 𝑓 𝑆𝑒𝑎𝑟𝑐ℎ is more complicated as a number of the implementations either do not provide a default or use the number of results to return (𝑘) as the value for 𝑒 𝑓 𝑆𝑒𝑎𝑟𝑐ℎ, thus it is use case dependant. As such, we set 𝑒 𝑓 𝑆𝑒𝑎𝑟𝑐ℎ as the median of the parameters observed in industry when 𝑘 for the given task is substituted into Table 1 . 
datasets: 
 In this section we describe the three main groups of datasets used in this study.In the synthetic case, we consider artificially generated vectors to control their properties, particularly their intrinsic dimensionality. To generate vectors of varying intrinsic dimensionality, we increase their complexity by varying the number of orthonormal basis vectors used in their construction. The Gram-Schmidt algorithm is used to create an orthonormal basis, and varying numbers of these orthonormal basis vectors are then combined to form datasets of vectors with varying intrinsic dimensionalities.Gram-Schmidt Orthonormalization. Let V = {v 1 , v 2 , . . . , v 𝑘 } be a set of 𝑘 randomly generated vectors in R 𝑑 , where 𝑑 represents the dimensionality of the space. The Gram-Schmidt process is applied to these vectors to obtain an orthonormal basis U = {u 1 , u 2 , . . . , u 𝑘 }, where each u 𝑖 is defined recursively by:u 𝑖 = w 𝑖 ∥w 𝑖 ∥ with w 𝑖 = v 𝑖 - 𝑖 -1 ∑︁ 𝑗=1 proj u 𝑗 (v 𝑖 )and the projection of v 𝑖 onto u 𝑗 is given by:proj u 𝑗 (v 𝑖 ) = ⟨v 𝑖 , u 𝑗 ⟩ ⟨u 𝑗 , u 𝑗 ⟩ u 𝑗Generating Data with Intrinsic Dimensionality of 𝑘. Once the orthonormal basis U is established, synthetic data X can be generated. This involves creating 𝑛 linear combinations of the basis vectors, where 𝑛 is the number of desired data vectors. We first define C, an 𝑛 × 𝑘 matrix whose entries 𝑐 𝑖 𝑗 are coefficients drawn from a normal distribution. Each data vector x 𝑖 is constructed as x 𝑖 = 𝑘 𝑗=1 𝑐 𝑖 𝑗 u 𝑗 . Thus, the data matrix X in R 𝑛×𝑑 is represented by X = CU where U is a 𝑘 ×𝑑 matrix containing the orthonormal basis vectors. Each row of X represents a data vector in the space spanned by the basis U. In practice this creates a dataset of 𝑛 unique random vectors with dimension 𝑑 which exist in a vector space of intrinsic dimensionality 𝑘.Text embedding models have been widely benchmarked for retrieval on a number of popular standard benchmark datasets. One popular aggregation of retrieval evaluations for text embedding models is the Massive Text Embedding Benchmark (MTEB) [30] . For this work we select a subset (see Table 2 ) of the datasets used for evaluation in MTEB as well as a selection of the most popular and best performing models at the time of this research.The datasets in Table 2 are used in this work. The datasets fall into one of two task types:• Asymmetric: The queries and corpus documents are asymmetric. Queries are questions or shorter statements used for retrieving related documents and answers from the corpus; • Symmetric: The queries and corpus documents are the same type of text. The goal is to find text in the corpus which is similar (for example, in the Quora dataset, the task is to find titles that are similar to the query title) For each of the datasets, embeddings are created with the models in Table 3 .Table 3 : Models used to embed the standard retrieval benchmark datasets.Embedding Dimension bge-base-en [50] 768 bge-small-en [50] 384 bge-base-en-v1.5 [50] 768 bge-small-en-v1.5 [50] 384 stella-base-en-v2 [2] 768 e5-base [46] 768 e5-small [46] 384 e5-base-v2 [46] 768 e5-small-v2 [46] 384 multilingual-e5-large [47] 1024 multilingual-e5-base [47] 768 multilingual-e5-small [47] 384 ember-v1 [36] 1024 all-MiniLM-L6-v2 [48] 384 bge-micro [3] 384 gte-base [26] 768The vector spaces produced for each model and dataset combination have their own properties regarding intrinsic dimensionality and local intrinsic dimensionality which are studied in this work.In addition to the synthetic data and standard benchmark retrieval datasets we also verify our findings under real-world conditions.Our real-world datasets are a proprietary collection of product images. We present two datasets:• An e-commerce catalogue of collectibles, handbags, streetwear, sneakers, and watches; and • A homewares catalogue of home, furniture, kitchenware, wall, renovation, bed, rugs, lighting, baby, lifestyle, pet, and office. To assess the applicability of our findings in a real-world setting we leverage the relationship between intrinsic dimensionality and categories in online retail catalogues. Items belonging to one category have their own intrinsic dimensionality which is lower than that of the entire dataset. To evaluate HNSW on this data, we use permutations of the categories to form insertion orders for data into the HNSW indexes; recall is computed for each order. 
evaluation methodology and results: 
 For the purposes of this work we define recall as the number of documents returned by an exact retriever which are also retrieved by an approximate one, in this case, KNN as the exact retriever and HNSW as the approximate retriever. Formally, for an approximate retriever (A) and an exact retriever (E) within a dataset 𝑋 using a set of queries 𝑄 where 𝑘 results are retrieved for each query, the recall at 𝑘 is defined as follows:Let 𝑅 𝐴 (𝑞, 𝑘) denote the set of 𝑘 results retrieved by the approximate retriever 𝐴 from 𝑋 for a query 𝑞 ∈ 𝑄, and 𝑅 𝐸 (𝑞, 𝑘) denote the set of 𝑘 results retrieved by the exact retriever 𝐸 from 𝑋 for the same query 𝑞. The recall for a single query 𝑞, is defined as the fraction of relevant documents retrieved by the approximate retriever 𝐴 out of the relevant documents retrieved by the exact retriever 𝐸. As described in section 4.1, the synthetic data consists of vectors of arbitrary intrinsic dimensionality which are created by combining varying numbers of orthonormal basis vectors. To assess the qualities of these vectors the intrinsic dimensionality is quantified with a Principal Component Analysis (PCA) based approach.Estimation of Intrinsic Dimensionality Using PCA. The intrinsic dimensionality of a dataset can be estimated using PCA by identifying the number of principal components that capture a significant proportion of the total variance in the dataset. Let 𝑋 be a dataset and 𝜆 𝑖 represent the explained variance ratio of the 𝑖-th principal component in the PCA. We compute a PCA on the dataset 𝑋 to obtain the explained variance ratios 𝜆 1 , 𝜆 2 , . . . , 𝜆 𝑛 , where 𝑛 is the total number of features in 𝑋 . The sum of the explained variance ratios for 𝑘 components is defined as 𝐶 (𝑘) = 𝑘 𝑖=1 𝜆 𝑖 . To determine the intrinsic dimensionality we find the smallest number 𝑘 such that the cumulative sum 𝐶 (𝑘) is greater than or equal to a pre-defined threshold 𝜃 (e.g., 𝜃 = 0.99 for 99% variance). The value of 𝑘 intrinsic represents the estimated intrinsic dimensionality of the dataset 𝑋 and describes the minimum dimensionality within the data which captures the specified proportion of variance 𝜃 in the data. For our work, 𝜃 = 0.99.Evaluation of Recall on Synthetic Data. We can verify the process used to generate this data by visualising the cumulative sum of explained variance ratio from the PCA for datasets constructed with varying numbers of orthonormal basis vectors as shown in Figure 1 . What we observe is that as the number of orthonormal basis vectors used to generate the synthetic data increases, the recall achieved with both HNSWLib and FAISS decreases. Figure 2 depicts the recall for HNSWLib and FAISS as the number of orthonormal basis vectors used to construct the data increases. The number of orthonormal basis vectors used to construct the data is the same for the indexed vectors and the query vectors. The synthetic data evaluation from section 5.1 shows that there exist properties of the vector space which can directly influence recall of HNSW for a given parameterisation. It follows that models whose vector spaces exhibit different properties for a given dataset can also impact recall. Many popular retrieval models are trained with some form of contrastive loss which provides no explicit control for properties such as the intrinsic dimensionality or local intrinsic dimensionality. Furthermore, training and evaluation of these models is typically only done in the context of exact KNN.Evaluation on benchmark datasets outlined in Table 2 for all models identified in Table 3 shows that rankings of models change when evaluated with various retrieval systems. This is to say that a retrieval leaderboard established with exact KNN is not perfectly representative of one produced using approximate nearest neighbours retrieval. Models are ranked using Normalised Discounted Cumulative Gain (NDCG) [23] . In Table 4 we observe that ranks shift up and down by up to three places when evaluated with different retrieval systems, these experiments use 𝑒 𝑓 𝑆𝑒𝑎𝑟𝑐ℎ = 10 at 𝑘 = 10. Smaller models like all-MiniLM-L6-V2 and bge-micro see improvements in relative performance when used in approximate retrieval systems, moving up the leader board by 2-3 places at 𝑒 𝑓 𝑆𝑒𝑎𝑟𝑐ℎ = 10.The analysis presented in section 5.1 reveals a significant relationship between the intrinsic dimensionality of the dataset and the recall performance of the system, with recall falling by approximately 50% as synthetic data approaches full rank. This observation leads us to hypothesise that HNSW graphs exhibit enhanced performance when they are structured in a manner that increases the probability of selecting entry points at each layer that are proximally located to any region within the graph. By proactively assessing the pointwise LID of data vectors, we can strategically influence the construction of the graph to optimise (or impair) its recall.In particular, constructing a graph with data sorted in descending order of LID appears to mimic a process similar to simulated annealing. This approach facilitates the late integration of clusters characterized by low LID values. Conversely, graphs initialized with data in ascending order of LID suffer from early establishment of tight localities in the graph comprised of low LID vectors. This setup deteriorates the initial conditions for graph optimization, leaving the integration of high LID vectors until the end stages. For the smaller datasets (arguana, nfcorpus, scifact, and scidocs) we are able to calculate the average path length of the final layer of the graphs constructed with HNSWLib. A Pearson correlation coefficient of 0.61 is observed between recall and average path length for these datasets across all models, this positive relationship indicates that longer path lengths yield better recall, this also aligns with the hypothesis that inserting high LID data first delays integration of tight clusters within the graph. The computation of pointwise LIDs was conducted using a Maximum Likelihood Estimation (MLE) method considering the 100 exact nearest neighbours [25] .Recall. The recall@10 was calculated for every model on the test sets of the 7 standard benchmark datasets identified in Table 2 , the recall for each of the models was then averaged across all datasets. Results are presented in Table 5 , bold values indicate the highest recall that was achieved for HN-SWLib or FAISS. When ordered by LID the recall is affected with consistent patterns, on average HNSWLib and FAISS implementations achieve 2.6 and 6.2 percentage points better recall respectively when data is inserted in descending LID order at 𝑒 𝑓 𝑆𝑒𝑎𝑟𝑐ℎ = 10.Table 5 shows that for all but one model (gte-base with FAISS) the recall was higher when inserting data in order of descending local intrinsic dimensionality. A random insertion order is consistently in between the ascending and descending LID insertion orders, reinforcing the hypothesis that the LID can be used to strategically influence the recall of the system, potentially towards an upper and lower extreme. The change in recall for each order varies significantly from the HNSWLib implementation to the FAISS implementation with a maximum difference between the two orders of 5.6 percentage points for HNSWLib (e5-small-v2) and 12.8 percentage points on FAISS (all-MiniLM-L6-v2). The same patterns are also observed at a higher 𝑒 𝑓 𝑆𝑒𝑎𝑟𝑐ℎ of 40 (Table 6 ).Table 6 shows that despite the overall higher recall, the differences between recall for each insertion order remain comparable. HNSWLib gives a maximum difference of 7.1 percentage points (e5small-v2) and FAISS gives a maximum difference of 11.5 percentage points (all-MiniLM-L6-v2) 5.2.3 LID Ordered Insertion and Relevance. Recall can be associated with other metrics which influence the efficacy of models on tasks such as information retrieval. Retrieval evaluation for all datasets using PyTREC Eval [41] shows a Pearson correlation coefficient of 0.71 is observed between recall@10 and NDCG@10. Table 7 shows implications in leaderboard ranking of different insertion orders for the HNSWLib implementation. The observations from Table 7 translate into impacts on downstream retrieval tasks. Given that model rankings shift under different insertion orders we can ascertain that each models vector space is not impacted equally; certain models exhibit more robustness to changes in insertion order.The insertion sequence of data, influenced by LID, represents a constructed scenario unlikely to mirror the more stochastic nature of real-world data indexing. Nonetheless, practical applications frequently encounter non-random data insertion phenomena. To elucidate the real-world relevance of insertion sequence and its correlation with LID, we examine data from two distinct online retail platforms -one focusing on fashion, the other on homewares.Contrastive Language-Image Pre-training (CLIP) [9, 19, 35, 39 ] models are used to generate embeddings from product images, with a series of GPT4 [31] generated e-commerce search terms serving as queries. Specifically the ViT-B-32 architecture with the laion2b_s34b_b79k checkpoint and the ViT-L-14 architecture with the laion2b_s32b_b82k checkpoint were used.Data was indexed sequentially organized by product categories as listed on the retailers' websites; data within each category is randomly ordered. This approach has similarities with LID-ordered insertion, presupposing that items within the same category exhibit closer proximity to one another than to items from disparate categories. We show that the category ordered insertions studied here have parallels to the LID ordered insertions in section 5.2.2. Analysis of the intrinsic dimensionality, as determined by the PCA method, shows that categories within the data have differing values to each other. Categories exhibit lower intrinsic dimensionalities than the dataset as a whole (Table 8 and Table 9 ).  
conclusion: 
 In this work we have shown that the construction of HNSW graphs can be sensitive to properties of the datasets and models utilised. The effect of insertion order for data into the graphs has real world impacts, especially in applications where the temporal component of incoming data is correlated with properties of the vector space that the data occupies; such as new product categories being added or domain shifts more generally.The relationship between intrinsic dimensionality and recall, paired with the relationship between recall and downstream retrieval tasks, indicates that optimal model selection for HNSW based retrieval systems is not as simple as following the results of a benchmark done with exact KNN.We hope that this work encourages further research into the HNSW algorithm to improve robustness against the insertion order of the data. Other advances may exist in model development as well, allowing for better understanding and control of properties of the vector space which can have impacts on recall in approximate retriever systems. 
 
 Appendix: The data can be accessed via Hugging Face here: https://huggingface.co/datasets/ anonymous/benchmark-embeddingsModern example include Lucene and Vespa which include production features like sharding and replicas.Parameters displayed here are current at time of writing, the parameters displayed at some cited URLs are subject to change with time. 


Decision: The paper presents an insightful examination of the influence of vector space dimensionality and the order of vector insertion on the recall performance of Hierarchical Navigable Small World (HNSW) Approximate Nearest Neighbor (ANN) algorithms. The authors utilize a synthetic task with controllable intrinsic dimensionality alongside retrieval tasks using both well-established and proprietary collections to thoroughly investigate these parameters. However, there are some weaknesses.

1. The evaluation only on the HNSW method, which descends the influence of this paper.
2. The evaluation of this paper is unconvinced. The evaluation results on MS MARCO should be conducted. More evaluation metrics should be conducted to show the performance of the paper.

Reviews: review: main_review: main_review: The article is well-written and demonstrates academic rigor, with mainly minor typographical errors. However, the disordered placement of references complicates the evaluation process. It is recommended that the authors arrange the references in a systematic order.
The research problem posed and addressed is compelling and could be helpful for researchers developing approximate nearest neighbor (ANN) algorithms or the broader community engaged in information retrieval. The article could enhance the community's understanding of parameter tuning in Hierarchical Navigable Small World (HNSW) while highlighting interesting observations regarding ordering and intrinsic dimensionality. Although the methodology employed is not particularly novel, some techniques such as the use of the average recall metric score against an exact retriever and the extensive experiments and discussions provided are likely to be highly valuable for theoretical research.
main_review: main_review: * The paper has creatively studied the performance of the HNSW algorithm under different intrinsic dimensions by constructing synthetic data. However, synthetic data may lack the complexity and diversity of real-world data. It could be questioned whether it is fair to conclude that the recall rate of HNSW decreases with the increase of intrinsic dimensions based solely on synthetic data.

* The paper has conducted studies on some real-world datasets, especially in the scenario of image retrieval. However, there are many other real-world retrieval datasets, such as code retrieval, etc. Exploring only the scenario of product image retrieval may be somewhat limited.

* HNSW is an algorithm that trades off spatial memory for higher efficiency and accuracy. I believe that research on the HNSW algorithm should focus more on how to reduce memory consumption without compromising performance.
main_review: main_review: The problem is relevant, and the experiments provide useful insights. My main concern is whether a Recall@10 is enough to fully assess the impact. The paper could use a re-organization as pages 6 and onwards were hard to follow.
main_review: main_review: * All figures and table titles are too short. Especially the titles of the tables should explain what the reader is supposed to see in the data. 
* Only HNSW and FAISS are used as ANN algorithms.
* While the paper is an easy read with a clear result, the work is also preliminary. The authors have pointed out, that the LID of the data generated by some modern DNN plays an important role in the ranking of the different ANN algorithm. However, only a few examples are shown. A more systematic approach (e.g, Transformer vs. CNNs vs. RNN, supervised vs. self-supervised learning) is desirable. Also, more about how to order incoming data during index construction should be further explored. Nevertheless, the paper is already inspiring in its current form.