Title: CRISP: Clustering Multi-Vector Representations for Denoising and Pruning
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: Multi-Vector Embedding, Late Interaction, Representation Pruning, Information Retrieval, Clustering

Abstract: Multi-vector models, such as ColBERT, are a significant advancement in neural information retrieval (IR), delivering state-of-the-art performance by representing queries and documents by multiple contextualized token-level embeddings. 
However, this increased representation size introduces considerable storage and computational overheads which have hindered widespread adoption in practice. A common approach to mitigate this overhead is to cluster the model's frozen vectors, but this strategy's effectiveness is fundamentally limited by the intrinsic clusterability of these embeddings.
 In this work, we introduce \crisp\ (Clustered Representations with Intrinsic Structure Pruning), a novel multi-vector training method which learns inherently clusterable representations directly within the end-to-end training process. By integrating clustering into the training phase rather than imposing it post-hoc, \crisp\ significantly outperforms post-hoc clustering at all representation sizes, as well as other token pruning methods.
 On the BEIR retrieval benchmarks, \crisp\ achieves a significant rate of \textbf{~3x} reduction in the number of vectors \textit{while outperforming} the original unpruned model. This indicates that learned clustering effectively denoises the model by filtering irrelevant information, thereby generating more robust multi-vector representations. With more aggressive clustering, \crisp\ achieves an \textbf{11x} reduction in the number of vectors with only a $3.6\%$ quality loss.

TL;DR: Multi-vector representation pruning for late-interaction retrieval models via a new clustering-based training method that produces fewer vectors with better quality.