Adaptive Semantic Prompt Caching with VectorQ
Authors: name: Luis Gaspar Schroeder, name: Shu Liu, name: Alejandro Cuadron, name: Mark Zhao, name: Stephan Krusche, name: Alfons Kemper, name: Matei Zaharia, name: Joseph E. Gonzalez

Semantic prompt caches reduce the latency and cost of large language model
(LLM) inference by reusing cached LLM-generated responses for semantically
similar prompts. Vector similarity metrics assign a numerical score to quantify
the similarity between an embedded prompt and its nearest neighbor in the
cache. Existing systems rely on a static threshold to classify whether the
similarity score is sufficiently high to result in a cache hit. We show that
this one-size-fits-all threshold is insufficient across different prompts. We
propose VectorQ, a framework to learn embedding-specific threshold regions that
adapt to the complexity and uncertainty of an embedding. Through evaluations on
a combination of four diverse datasets, we show that VectorQ consistently
outperforms state-of-the-art systems across all static thresholds, achieving up
to 12x increases in cache hit rate and error rate reductions up to 92%.