SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Distributed-based loss: Instead of being oriented by the score between query and candidates as above, we attempt to minimize the distance between the queries and top-k neighbors by calculating the similarity scores with all the centroids. Thus, we could obtain more information from centroids and focus on the top-K nearest neighbors. The distributed-based loss function follows as:
$$
\mathcal {L} (\boldsymbol {q}, \mathcal {D} _ {K} ^ {T}; \boldsymbol {C}) = - \sum_ {i \in \mathcal {D} _ {K} ^ {T}} \sum_ {b = 1} ^ {B} \sum_ {k = 1} ^ {W} \tilde {p} _ {b k} ^ {q} \log \left(\tilde {p} _ {b k} ^ {d _ {i}} \cdot w _ {i}\right) \tag {6}
$$
This loss requires the enumeration of all the centroids, while the Weighted KL loss only includes parts of the centroids corresponding to the quantized function. It also aligns with the goal of nearest searching for the query with the learnable centroids as the bridge.
C.2 Experimental Performances
We compare the effectiveness of the three different distillation losses, i.e., Weighted KL loss, Distributed-based loss, and Lambda rank loss as reported in Table 8.
Findings. Overall, the Distributed-based loss leads to comparatively better performances than Weighted KL loss and Lambdarank loss. Compared with Weighted KL Loss and Lambdarank Loss, Distributed-based Loss gains the  $2.03\%$  and  $3.54\%$  improvements of Recall@10,  $0.38\%$ , and  $0.70\%$  of NDCG@10, respectively. The Lambdarank Loss concerns more about the relationships between the pair of items, while the other two care about the whole ranking order of the list. The weighted KL loss actually optimizes parts of the centroids, depending on which query vectors and candidate vectors are quantized, to match the ranking list, while all of the centroids are updated in the Distributed-based loss since the probabilities are calculated over all the centroids. Furthermore, the Distributed-based loss requires the similarity calculation between the original input vectors and the centroids, which eliminates the error caused by the compressed functions. The last observation is that Distributed-based Loss works better on inner product metric datasets, since it obtains the average improvements of  $2.24\%$  and  $3.33\%$  of Recall@10 on L2 distance and inner product, respectively, wherein the overall improvements for inner-product similarities.
Table 8: The results of KDindex under different distillation loss functions.
Loss Function	SIFT1M	GIST1M	MS MARCO Doc	MS MARCO Passage
Recall@10	NDCG@10	Recall@10	NDCG@10	Recall@10	MRR@10	Recall@10	MRR@10
LambdaRank Loss	36.32	79.18	20.69	62.06	17.60	40.01	11.10	34.86
Weighted KL Loss	36.68	79.33	21.02	62.75	18.24	40.93	11.06	34.63
Distributed-based Loss	37.30	80.01	21.33	63.17	18.93	41.69	11.19	35.23
D Search Efficiency and Retrieval Quality
The search efficiency and the comprise performance are important indicators of a compression search index. Thus, we report the effectiveness-latency performance for inference in Figure 4.
The time efficiency is measured by the latency of getting the Top-10 candidates for each query. The retrieval performance and the running time are governed by the number of posting lists we probe among the codebooks. The compressed ANNS methods (ScaNN and BLISS) and an uncompressed method (HNSW) are introduced for comparison. We can see that KDindex substantially outper
Figure 4: Comparison with more compressed ANNS models and uncompressed method (HNSW, ngtqg, qsgngt, vamana, and Milvus) in terms of effectiveness-latency trade-off on two datasets. Up and left is better. The results of ScaNN, HNSW, ngtqg, qsgngt, vamana, and Milvus come from ANN-Benchmarks [1].
forms baselines in terms of both effectiveness and time efficiency. Specifically, we find that KDindex cost less time to arrive at a similar Recall than both ScaNN and BLISS. Although HNSW can achieve better results in the early stage, KDindex will surpass it after half a second.
E Limitations and Future Works
In this paper, we propose a novel knowledge distillation framework for high dimension index, which reduce storage obviously and can learn neighbor information from the teacher model. Especially, KDindex is independent to label (such as interaction information in the recommendation system or ground-truth neighbors in ANNS), which makes it flexible to be applied in more label-free scenarios. In the future, we will try more student models such as lattice quantization, whose codes already imply neighbors relationship. And we will take labels into account to improve retrieval performance progressively. We will further improve our work to benefit the broad community.