SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Twin-tower representation model. We use T5-ANCE [36] to generate embeddings for RQ clustering. T5-ANCE is an effective embedding-based retrieval model which replaces the backbones of ANCE [51] with T5 [39]. We use T5-ANCE and AR2 [54] to generate embeddings for ensembling. AR2 is the state-of-the-art twin tower retrieval model. These two methods have different backbone models, indicating that our method can be applied to different twin-tower representation models. To obtain the search result from embedding-based retrieval, we adopt HNSW [30], a novel ANN search technique implemented in Faiss [20], with 256 neighbors per vertex. We have also tried more neighbors per vertex but got no gain compared to 256.
Ensemble. All the ensemble strategies require searching among the top-k clusters retrieved by the sequence-to-sequence model. Instead of performing ANN search as in traditional embedding-based retrieval, we employ brute-force search in top-k clusters. This is because the number of candidates in the top-k clusters is relatively small, typically on the order of ten thousand, given a beam size of 1000. The hyperparameters of the ensemble are determined through grid-search. We study the impact of hyperparameters in Section 4.8.
Baselines. We compare our method with sparse retrieval methods BM25 [42] and SPLADE [10], dense retrieval methods T5-ANCE and AR2 with ANN, and model-based retrieval NCI. The comparisons with T5-ANCE/AR2 and NCI also reflect the gain of ensemble strategies. For NCI, we build a hierarchical tree with a depth of 8. The other hyperparameters of the baseline methods remain the same as the original papers.
4.3 Results on MSMARCO Passage
Table 2 presents the retrieval results on MSMARCO Passage dataset. We compare different ensemble strategies of MEVI with baselines. We report the results of searches in the top-10, top-100, and top-1000 clusters, and the ensemble of these results with the embedding-based retrieval methods. Among all the tested methods, the ensemble of embedding-based retrieval and generation-based retrieval outperforms all baselines. Compared to the ANN baseline, MEVI improves  $3.62\% / 2.62\%$  for MRR@10,  $7.32\% / 5.43\%$  for Recall@50,  $10.54\% / 8.00\%$  for Recall@1000 on AR2 / T5-ANCE respectively. The more clusters to be considered, the better the retrieval performance. Even without ensemble, vanilla MEVI performs better than dense retrieval methods when considering top-1000 clusters. After ensemble, MEVI can achieve better results than ANN with only top-10 clusters, which shows that the ensemble combines the advantages of ANN and top-k clusters, making them complement each other. Compared to NCI, MEVI performs better especially on MRR@10, because MEVI utilizes generation-based retrieval with smaller decoder steps, allowing the model to learn a better ranking.
Table 2: Experiment results on MSMARCO Passage (Dev).
Method	MRR@10	R@50	R@1000
BM25	18.7	59.2	85.7
SPLADE	32.2	/	95.5
T5-ANCE (HNSW)	33.21	77.30	88.61
AR2 (HNSW)	35.54	78.80	87.11
NCI	26.18	74.68	92.44
MEVI Top-10 Clus	32.05	63.25	66.82
MEVI Top-100 Clus	35.16	79.14	88.22
MEVI Top-1000 Clus	35.76	82.37	95.17
MEVI Top-10 Clus & T5-ANCE (HNSW)	35.22	81.29	93.21
MEVI Top-100 Clus & T5-ANCE (HNSW)	35.60	82.27	95.62
MEVI Top-1000 Clus & T5-ANCE (HNSW)	35.83	82.73	96.61
MEVI Top-10 Clus & AR2 (HNSW)	37.00	82.64	93.46
MEVI Top-100 Clus & AR2 (HNSW)	38.42	84.52	96.23
MEVI Top-1000 Clus & AR2 (HNSW)	39.16	86.12	97.65
4.4 Results on Natural Questions
Table 3 presents the retrieval results on Natural Questions dataset. Compared to the ANN baseline, MEVI improves  $5.04\%$  for Recall@5,  $5.46\%$  for Recall@20,  $5.96\%$  for Recall@100 on AR2 model. Vanilla MEVI also performs better than AR2 (HNSW) when searching among top-1000 clusters. After ensemble, MEVI can outperform AR2 (HNSW) with only 10 clusters.
Table 3: Experiment results on Natural Questions (Test).
Method	R@5	R@20	R@100
BM25	/	59.1	73.7
AR2 (HNSW)	70.89	78.50	83.02
MEVI Top-10 Clus	59.61	66.45	71.63
MEVI Top-100 Clus	70.33	77.23	81.77
MEVI Top-1000 Clus	75.57	82.83	87.31
MEVI Top-10 Clus & AR2 (HNSW)	74.10	82.11	86.43
MEVI Top-100 Clus & AR2 (HNSW)	74.43	82.71	87.51
MEVI Top-1000 Clus & AR2 (HNSW)	75.93	83.96	88.98
4.5 Dynamic update of documents
In order to test whether MEVI can support the dynamic update of documents, we randomly remove  $10\%$  of the documents in the MSMARCO Passage dataset during training, and add these  $10\%$  documents back during evaluation. We compare MEVI with the embedding-based retrieval method T5-ANCE. Since NCI only maintains a fixed document tree that does not support dynamic update of documents, we do not include NCI in this experiment.
Table 4 shows the results. Performance drop compared to normal training are listed in the parentheses. All tested methods lose some model accuracy when  $10\%$  of the documents are discarded during training. MEVI still performs well compared to ANN, showing the ability to handle unseen documents. From experiments we also find that the recall drops less as more clusters are retrieved, because more clusters provide more possible semantics information about unseen documents.
Table 4: Train with random  ${90}\%$  documents,and evaluate with all documents.
Method	MRR@10	R@50	R@1000
T5-ANCE (HNSW)	29.34 (3.87)	73.09 (4.21)	86.26 (2.35)
MEVI Top-10 Clus	28.20 (3.85)	55.43 (7.82)	58.81 (8.01)
MEVI Top-100 Clus	33.48 (1.68)	74.16 (4.98)	82.68 (5.54)
MEVI Top-1000 Clus	35.38 (0.38)	81.06 (1.31)	93.40 (1.77)
4.6 Residual quantization
In this section, we first analyze the advantages of RQ over ordinary KMeans clustering, and then study the impact of RQ codebook configuration.
We choose RQ instead of normal hierarchical KMeans in MEVI. Compared to hierarchical KMeans that focuses on more fine-grained clustering within large clusters in each layer, RQ adopts residual information to address the errors of the previous layer, making the clustering results more precise and robust. We conduct an experiment on MSMARCO Passage to compare these two algorithms in MEVI. In the experiments, we set layer depth to 4 and the number of centroids per layer to 32. In Table 5, RQ generally achieves better recall than normal KMeans with the same number of clusters.
In Table 6, we use  $RQ(x \times y)$  to represent an RQ codebook with  $x$  layers,  $y$  bits ( $2^{y}$  centroids) per layer, and a total of  $2^{xy}$  clusters. For simplicity, we train the models on MSMARCO Passage with only 1 generated query per document for augmentation. We report MRR@10, Recall@100, the total recall of top-k clusters, and the average number of retrieved documents per query. The numbers of
Table 5: Comparison of KMeans and RQ in MEVI.
Method	MRR@10	R@50	R@1000
MEVI-KMeans Top-10 Clus	31.62	65.09	68.25
MEVI-KMeans Top-100 Clus	34.82	77.30	87.93
MEVI-KMeans Top-1000 Clus	35.65	81.01	94.17
MEVI-RQ Top-10 Clus	32.05	63.25	66.82
MEVI-RQ Top-100 Clus	35.16	79.14	88.22
MEVI-RQ Top-1000 Clus	35.76	82.37	95.17
layers and bits both determine the size of cluster-id space, while the number of retrieved clusters  $k$  determines the number of retrieved documents per query. To fairly compare these configurations, we would like to align the number of retrieved documents per query to the same order of magnitude. From  $RQ(3 \times 4)$  to  $RQ(5 \times 5)$ , the number of clusters increases, and the number of documents in each cluster decreases, so we enlarge the number of clusters from 3 to 1000 to align the number of the documents per query. In general, the larger the number of candidate documents, the more likely the model is to recall the correct document. With the same order of magnitude of retrieved documents,  $RQ(4 \times 5)$  performs well with moderate number of retrieved clusters. Although  $RQ(5 \times 5)$  performs better in this case, it needs to retrieve more clusters to maintain an appropriate number of document candidates, which poses serious memory challenges. Larger configurations also mean more decoder steps, resulting in larger inference latency. Hence for MSMARCO Passage dataset,  $RQ(4 \times 5)$  is more proper. We leave the support for larger RQ in future work.
Table 6: Model performance with different RQ configurations.
Configuration	# Clusters	MRR@10	R@100	R@Clusters	#Docs / Query
\( RQ(3 \times 4) \)	3	26.38	55.05	58.83	13318
\( RQ(4 \times 4) \)	10	29.15	63.28	67.28	9750
\( RQ(4 \times 5) \)	100	33.36	76.14	80.71	4972
\( RQ(5 \times 4) \)	100	32.57	72.92	77.11	4764
\( RQ(5 \times 5) \)	1000	34.12	79.69	85.62	3999
4.7 Efficiency
We test the online serving latency of MEVI on a single A6000 card. Since the inference time is affected by the number of retrieved clusters, we list the latency of different settings in Table 7. For MEVI and NCI, the latency mainly comes from the stepwise generation of the autoregressive sequence-to-sequence model. Latency increases with the number of clusters, because larger beam size leads to larger autoregressive generation time, as well as larger number of candidates to be searched. NCI has much longer decoder steps than MEVI, resulting in a significant increase in latency. Latency of ANN includes the query encoding time and the HNSW searching time; it is the fastest method, which is appealing to real applications. Compared to ANN, MEVI can achieve a good performance and latency trade-off when  $k = 10$ , improving the MRR@10 metric by more than 2 absolute points with a less than 100ms latency. This is acceptable online, as by adopting some acceleration techniques (e.g., FP16, ONNX Runtime [7]), we can further improve the average serving latency to less than 40ms.
Table 7: Trade-off between serving latency and model performance.
Method	MRR@10	Latency (ms)
T5-ANCE (HNSW)	33.21	19.71
NCI	26.18	2899.17
MEVI Top-10 & T5-ANCE (HNSW)	35.22	96.87
MEVI Top-100 & T5-ANCE (HNSW)	35.60	222.55
MEVI Top-1000 & T5-ANCE (HNSW)	35.83	1662.84
4.8 Hyperparameters
We try different  $K$  for Recall and MRR on MSMARCO Passage in Figure 2. "HNSW" means T5-ANCE with HNSW index, "Ensemble" means the ensemble result of T5-ANCE (HNSW) and MEVI Top-1000 Cluster. MEVI ensemble results consistently outperform T5-ANCE (HNSW).
In the ensemble process, we search the values of hyperparameters  $\alpha$  and  $\beta$  in a proper range and choose the configuration with the best model metrics. Since the ensemble process does not incur additional training and inference cost, the time for grid search can be ignored. Figure 3 shows the MRR@10 of MEVI Top-1000 & T5-ANCE (HNSW) on MSMARCO Passage with different  $\alpha$  and  $\beta$ . The optimal configuration fully leverages the information of both components. Our ensemble method is also robust to hyperparameters, as the ensemble result outperforms the baseline (MEVI Top-1000 Clus 35.76) when  $0.3 \leq \alpha \leq 0.7$  and  $0.01 \leq \beta \leq 0.03$ .
(a) Recall@K.
Figure 2: Recall and MRR at different K.
(b) MRR@K.
Figure 3: Impact of hyperparameters.
(a) MRR@10 v.s.  $\alpha$
(b) MRR@10 v.s.  $\beta$
5 Limitations & future works
Despite the significant contribution of MEVI, there are opportunities for further improvements to enhance the model performance. First, the twin-tower model and the sequence-to-sequence model are not jointly learned, which may not yield the optimal retrieval performance. Second, the capacity of the sequence-to-sequence model is still insufficient to cope with large-scale corpus within acceptable inference latency. In future works, these issues may be addressed by the following techniques: (a) Joint-training of RQ codebooks and two models can be explored [53, 48]. (b) Model compression and acceleration techniques can be applied, such as knowledge distillation [16], non-autoregressive generation [50].
In this work, we introduce a novel Model-enhanced Vector Index (MEVI), which combines the advantages of both the sequence-to-sequence autoregressive model and the twin-tower dense representation model. It can be effectively applied in real-world applications due to its ability to achieve both high recall performance and fast retrieval speed on large-scale corpus. MEVI constructs an RQ structure to hierarchically cluster large-scale documents, enabling the sequence-to-sequence model to directly generate the relevant cluster identifiers given an input query; the retrieved candidate documents in the top-k clusters are further ensembled with embedding-based retrieval results for candidates re-ranking. We empirically show that MEVI achieves better model performance than baselines on the widely used large-scale retrieval datasets MSMARCO Passage and Natural Questions.
This work is supported by National Key R&D Program of China (2022ZD0116315) and National Natural Science Foundation of China (61832001 and U22B2037). We thank Chenyan Xiong for his participation in the early discussions of MEVI and his guidance on the use of the T5-ANCE model. We thank Hang Zhang and Yeyun Gong for their guidance on the use of the AR2 model.
[1] Jon Louis Bentley. Multidimensional binary search trees used for associative searching. Communications of the ACM, 18(9):509-517, 1975.
[2] Michele Bevilacqua, Giuseppe Ottaviano, Patrick S. H. Lewis, Scott Yih, Sebastian Riedel, and Fabio Petroni. Autoregressive search engines: Generating substrings as document identifiers. In Advances in Neural Information Processing Systems 35 (NeurIPS), 2022.
[3] Qi Chen, Bing Zhao, Haidong Wang, Mingqin Li, Chuanjie Liu, Zengzhong Li, Mao Yang, and Jingdong Wang. SPANN: highly-efficient billion-scale approximate nearest neighborhood search. In Advances in Neural Information Processing Systems 34 (NeurIPS), 2021.
[4] Zhuyun Dai and Jamie Callan. Context-aware sentence/passage term importance estimation for first stage retrieval. CoRR, abs/1910.10687, 2019.
[5] Fu-Rong Dang, Jintao Tang, Kunyuan Pang, Ting Wang, Sha-Sha Li, and Xiao Li. Constructing an educational knowledge graph with concepts linked to wikipedia. Journal of Computer Science and Technology, 36(5):1200-1211, 2021.
[6] Mayur Datar, Nicole Immorlica, Piotr Indyk, and Vahab S. Mirrokni. Locality-sensitive hashing scheme based on p-stable distributions. In Proceedings of the 20th ACM Symposium on Computational Geometry, 2004.
[7] ONNX Runtime developers. Onnx runtime. https://onnxruntime.ai/, 2021.
[8] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT, 2019.
[9] Qian Dong, Shuzi Niu, Tao Yuan, and Yucheng Li. Disentangled graph recurrent network for document ranking. Data Science and Engineering, 7(1):30-43, 2022.
[10] Thibault Formal, Benjamin Piwowarski, and Stéphane Clinchant. SPLADE: sparse lexical and expansion model for first stage ranking. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, 2021.
[11] Luyu Gao and Jamie Callan. Condenser: a pre-training architecture for dense retrieval. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP, 2021.
[12] Luyu Gao and Jamie Callan. Unsupervised corpus aware language model pre-training for dense passage retrieval. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, ACL, 2022.
[13] Jiafeng Guo, Yixing Fan, Qingyao Ai, and W. Bruce Croft. A deep relevance matching model for ad-hoc retrieval. In Proceedings of the 25th ACM International Conference on Information and Knowledge Management, CIKM, 2016.
[14] Tonglei Guo, Jiafeng Guo, Yixing Fan, Yanyan Lan, Jun Xu, and Xueqi Cheng. A comparison between term-based and embedding-based methods for initial retrieval. In Information Retrieval - 24th China Conference, CCIR, 2018.
[15] John A Hartigan and Manchek A Wong. A k-means clustering algorithm. Journal of the royal statistical society. series c (applied statistics), 28(1):100-108, 1979.
[47] Zhijing Wu, Yiqun Liu, Jiaxin Mao, Min Zhang, and Shaoping Ma. Leveraging document-level and query-level passage cumulative gain for document ranking. Journal of Computer Science and Technology, 37(4):814-838, 2022.
[48] Shitao Xiao, Zheng Liu, Yingxia Shao, Defu Lian, and Xing Xie. Matching-oriented embedding quantization for ad-hoc retrieval. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP, 2021.
[49] Shitao Xiao, Zheng Liu, Weihao Han, Jianjin Zhang, Defu Lian, Yeyun Gong, Qi Chen, Fan Yang, Hao Sun, Yingxia Shao, and Xing Xie. Distill-vq: Learning retrieval oriented vector quantization by distilling knowledge from dense embeddings. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, 2022.
[50] Yisheng Xiao, Lijun Wu, Junliang Guo, Juntao Li, Min Zhang, Tao Qin, and Tie-Yan Liu. A survey on non-autoregressive generation for neural machine translation and beyond. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(10):11407-11427, 2023.
[51] Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul N. Bennett, Junaid Ahmed, and Arnold Overwijk. Approximate nearest neighbor negative contrastive learning for dense text retrieval. In The 9th International Conference on Learning Representations, ICLR, 2021.
[52] Wei Yang, Haotian Zhang, and Jimmy Lin. Simple applications of BERT for ad hoc document retrieval. CoRR, abs/1903.10972, 2019.
[53] Han Zhang, Hongwei Shen, Yiming Qiu, Yunjiang Jiang, Songlin Wang, Sulong Xu, Yun Xiao, Bo Long, and Wen-Yun Yang. Joint learning of deep retrieval model and product quantization based embedding index. In The 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, 2021.
[54] Hang Zhang, Yeyun Gong, Yelong Shen, Jiancheng Lv, Nan Duan, and Weizhu Chen. Adversarial retriever-ranker for dense text retrieval. In The 10th International Conference on Learning Representations, ICLR, 2022.
[55] Mingdong Zhu, Derong Shen, Lixin Xu, and Xianfang Wang. Scalable multi-grained cross-modal similarity query with interpretability. Data Science and Engineering, 6(3):280-293, 2021.
[56] Shengyao Zhuang, Hang Li, and Guido Zuccon. Deep query likelihood model for information retrieval. In Advances in Information Retrieval - 43rd European Conference on IR Research, ECIR, 2021.
Appendices for Model-enhanced Vector Index
A Application in industry
We apply MEVI in a commercial advertising system. When users submit their queries, the system will retrieve the advertisements from the corpus. A successful retrieval algorithm should ensure that the retrieved advertisements satisfy two essential conditions. First, they should have a relatively high CTR (click-through rate) to maximize the number of clicks. Second, they should have strong relevance to the user's query, ensuring a good user experience that the displayed advertisements are closely related to the user intent.
Dataset and training details. In this scenario, a large-scale internal dataset is selected from the historical advertising logs for offline evaluation. We collect approximately 80 million query-ad pairs for training and 40 million for testing. There are approximately 20 million advertisements in the sampled dataset. To satisfy both the CTR and relevance goals, the retrieval model is followed by a relevance-targeting re-rank model to select top ads from the recall candidates. We adopt RQ codebook with 5 clustering layers and 5 bits per layer for MEVI. The model is trained on 8 V100 GPU cards for 2 weeks, with a batch size of 256. The evaluation beam size is 10. Other settings and hyperparameters keep the same as the MSMARCO Passage experiments.
Metrics. We adopt several metrics to evaluate recall and relevance performance. Hitrate (Hit@K) measures how many clicked ads are included in top K results given a query. MRR (MRR@K) measures the ranking performance of the ads. Average Relevance score (Rel@K) measures the relevance score from top-K results, calculated by a commercial large language model finetuned by human labels, which has been proved to be effective and correlate closely to human labels.
Baseline. We adopt a commercial embedding-based retrieval (EBR) methods as the baseline, which has 4 transformer layers on both query and document sides. The hidden dimension is 768 and the number of head is 12. The EBR model is trained on 8 V100 GPU cards for 6 epochs. After the training finished, we generate query and ads embeddings and adopt an efficient ANN algorithm used in production to search for relevant advertisements. We also use the same relevance model to rerank the recall candidates.
Table 1: Experiment results on commercial advertising dataset.
Method	Hit@50	Hit@500	MRR@50	MRR@500	Rel@50	Rel@500
EBR	61.80	95.23	73.42	97.07	39.61	22.92
Top-10 Clusters	69.62	83.40	75.30	84.35	26.57	14.98
MEVI	71.92	96.25	80.55	97.75	39.78	23.23
Results. The experimental results are shown in Table 1. The search results within top-10 clusters demonstrate strong recall performance for the top 50 results. However, the top 500 results show a decline in performance compared to the EBR baseline, primarily due to the beam size being limited to 10 during MEVI inference, which results in a smaller search space. Increasing the beam size could potentially improve performance but would also place a greater computation burden on the relevance model. Consequently, a beam size of 10 was ultimately chosen for production scenario.
Despite these limitations, the ensemble results reveal a substantial increase across all metrics, indicating that MEVI serves as an excellent candidate provider for recall and is complementary to embedding-based retrieval methods, which is meaningful for multiple-recall scenarios. By incorporating MEVI, the commercial advertising system can significantly enhance both recall and relevance performance.
Online A/B test. MEVI also demonstrates impressive results in online A/B test. It achieves a  $0.71\%$  RPM (Revenue per Mille searches) growth in the A/B test with a P-Value of 0.00021 (a P-Value less than 0.05 is considered statistically significant in production). Moreover, MEVI shows a  $0.60\%$  reduction of bad relevance cases according to the human judgement, further demonstrating that MEVI not only boost advertising revenue, but also effectively improve user experiences with better query-ad relevances.