ColBERT-serve: Efficient Multi-Stage Memory-Mapped Scoring
Authors: Kaili Huang, Thejas Venkatesh, Uma Dingankar, Antonio Mallia, Daniel F Campos, Christopher Potts, Matei Zaharia, Kwabena Boahen, Keshav Santhanam, Saarthak Sarup, Omar Khattab


Reviewer paper summaries: The paper proposes a system named ColBERT-serve that utilizes memory-mapping and a multi-stage retrieval process to enable the deployment of state-of-the-art models, specifically ColBERTv2, on resource-constrained machines. The system can significantly reduce RAM requirements and latency under varying traffic conditions. The paper presents an evaluation of the proposed system on multiple datasets, demonstrating its effectiveness in terms of retrieval quality and latency.
Summary of strengths: 1.	This paper follows an interesting idea and focuses on deploying sophisticated IR models under the low-resource settings.
2.	The proposed method shows a significant reduction in RAM requirements through memory-mapping compared to full ColBERTv2.
3.	The authors provide thorough evaluations of memory-mapped index sizes, retrieval quality, and latency under varying traffic conditions, demonstrating the system's advantages.
Summary of weaknesses: 1.	The writing of the paper needs further improvement, and some parts are confusing. For example, line 130, what does “MMAP SPLADEv2” mean, do you apply memory-mapping for SPLADEv2?
2.	The latency of end-to-end retrieval (i.e., MMAP ColBERTv2) is high and only when used in reranking the proposed memory-mapping method has advantages in latency, which may limit the application of this method. 
3.	The experiments on the Lotte dataset do not seem to be directly related to the problem that needs to be addressed in the paper, since the dataset is small enough to fit into a machine with restricted memory and the memory-mapping technique is not used.
Comments, suggestions and typos: 1.	It would be interesting to explore how to further optimize the efficiency when applying memory-mapping to the end-to-end retrieval.
2.	The experiments on the Lotte dataset need more explanations. For example, why is this experiment retained even without using memory-mapping techniques, and why only choose the lifestyle dataset instead of all lotte datasets?
Overall assessment scores: 3.5