ICML 2025

Auditing Prompt Caching in Language Model APIs
Chenchen Gu 1 Xiang Lisa Li 1 Rohith Kuditipudi 1 Percy Liang 1 Tatsunori Hashimoto 1
Abstract
Prompt caching in large language models (LLMs)
results in data-dependent timing variations:
cached prompts are processed faster than non-
cached prompts. These timing differences intro-
duce the risk of side-channel timing attacks. For
example, if the cache is shared across users, an
attacker could identify cached prompts from fast
API response times to learn information about
other users’ prompts. Because prompt caching
may cause privacy leakage, transparency around
the caching policies of API providers is important.
To this end, we develop and conduct statistical au-
dits to detect prompt caching in real-world LLM
API providers. We detect global cache sharing
across users in seven API providers, including
OpenAI, resulting in potential privacy leakage
about users’ prompts. Timing variations due to
prompt caching can also result in leakage of in-
formation about model architecture. Namely, we
find evidence that OpenAI’s embedding model
is a decoder-only Transformer, which was previ-
ously not publicly known.1
1. Introduction
Transformer large language models (LLMs) are computa-
tionally expensive and slow to run. To address this challenge,
recent work has developed optimizations to make LLM in-
ference and serving more efficient, such as prompt caching
(Zheng et al., 2024a; Gim et al., 2024). In prompt caching,
reuse of the attention key-value (KV) cache across requests
results in cache hits and faster response times for prompts
that share a prefix with a cached prompt.
However, prompt caching results in data-dependent timing
1Stanford University.
Correspondence to:
Chenchen
Gu
<cygu@cs.stanford.edu>,
Tatsunori
Hashimoto
<thashim@stanford.edu>.
Proceedings of the 42 nd International Conference on Machine
Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025
by the author(s).
1We release code and data at https://github.c...