Adaptive Semantic Prompt Caching with VectorQ
Authors: name: Luis Gaspar Schroeder; name: Shu Liu; name: Alejandro Cuadron; name: Mark Zhao; name: Stephan Krusche; name: Alfons Kemper; name: Matei Zaharia; name: Joseph E. Gonzalez
Published: 2025-02-06 04:16:20

Abstract: Semantic prompt caches reduce the latency and cost of large language model
(LLM) inference by reusing cached LLM-generated responses for semantically
similar prompts. Vector similarity metrics assign a numerical score to quantify
the similarity between an embedded prompt and its nearest neighbor in the
cache. Existing systems rely on a static threshold to classify whether the
similarity score is sufficiently high to result in a cache hit. We show that
this one-size-fits-all threshold is insufficient across different prompts. We
propose VectorQ, a framework to learn embedding-specific threshold regions that
adapt to the complexity and uncertainty of an embedding. Through evaluations on
a combination of four diverse datasets, we show that VectorQ consistently
outperforms state-of-the-art systems across all static thresholds, achieving up
to 12x increases in cache hit rate and error rate reductions up to 92%.