Auditing Prompt Caching in Language Model APIs
Authors: Chenchen Gu
Xiang Lisa Li
Rohith Kuditipudi
Percy Liang
Tatsunori Hashimoto
Venue: ICML 2025 poster (2025)
Keywords: audit
prompt caching
privacy
transparency
large language models
Primary area: social_aspects->accountability_transparency_and_interpretability
TL;DR: We develop and conduct statistical audits to detect prompt caching and cache sharing in real-world language model APIs, as prompt caching may cause privacy leakage.

Abstract: Prompt caching in large language models (LLMs) results in data-dependent timing variations: cached prompts are processed faster than non-cached prompts. These timing differences introduce the risk of side-channel timing attacks. For example, if the cache is shared across users, an attacker could identify cached prompts from fast API response times to learn information about other users' prompts. Because prompt caching may cause privacy leakage, transparency around the caching policies of API providers is important. To this end, we develop and conduct statistical audits to detect prompt caching in real-world LLM API providers. We detect global cache sharing across users in seven API providers, including OpenAI, resulting in potential privacy leakage about users' prompts. Timing variations due to prompt caching can also result in leakage of information about model architecture. Namely, we find evidence that OpenAI's embedding model is a decoder-only Transformer, which was previously not publicly known.

Auditing Prompt Caching in Language Model APIs
Chenchen Gu Xiang Lisa Li
Rohith Kuditipudi
Perey Liang'
Tatsunori Hashimoto'
Abstract
(1) Prompt becomes cached
Cache miss slow response time
(3) Cache = fast response time
Prompt caching in large language models (LLMs) results in data-dependent timing variations: cached prompts
Victim
Attacker
Attacker
processed faster than non- cached prompts. These timing differences intro- duce the risk of side-channel timing attacks. For example, if the cache is shared across users, an attacker could identify cached prompts from fast API response times to learn information about other users' prompts. Because prompt caching may cause privacy leakage, transparency around the caching policies of API providers is important. To this end, we develop and conduct statistical au- dits to detect prompt caching in real-world LLM API providers. We detect global cache sharing across users in seven API providers, including OpenAI, resulting potential privacy leakage about users' prompts. Timing variations due prompt caching can also result in leakage of in- formation about model architecture. Namely, we find evidence that OpenAl's embedding model is decoder-only Transformer, which was previ- ously not publicly known."
The guick brown fox jumps
Loremn jesum diblor amet
The quilii
0.5
brown fox
0.15
API
API
Prompt Cache
Prompt Cache
Prompt Cache
quibk brion for jurtus
The quack Brown jumtsupps
Ipouth
Figure I. An example illustrating prompt caching. (1) A vietim sends a prompt the API. which then becomes cached. (2) An attacker sends a new prompt, resulting in a cache miss and slow response time. (3) An attacker sends prompt that shares prefix with the victim's 's prompt, resulting in cache hit. From the fast response time, the attacker can infer that cache hit occurred, which potentially reveals information about other users' prompts.
variations-cacher prompts will be processed faster than non-cached prompts, introducing the risk of side-channel timing attacks and information leakage. In particular, an attacker could identify prompts that yield fast API response times; such prompts are likely cached. If the cache shared across users, then a prompt being cached implies that other user recently sent that prompt. Figure illustrates an example of prompt caching and potential privacy leakage. In general, timing differences between cache hits and cache misses have been widely exploited in computer security, such as the infamous Meltdown (Lipp et al., 2018) and Spectre attacks (Kocher al., 2019).
1. Introduction
Transformer large language models (LLMs)) are computa- tionally expensive and slow run. To address this challenge, recent work has developed optimizations make LLM in- ference and serving more efficient, such as prompt caching (Zheng et al., 2024a; Gim et al., 2024). In prompt caching, reuse of the attention key-value (KV) cache across requests results in cache hits and faster response times for prompts that share prefix with cached prompt.
Because prompt caching may result in privacy leakage, it is important for users to know about the prompt caching policies of API providers. Some API providers have announced that they perform prompt caching, such as Anthropic (2024b) and OpenAl (2024b), but other API providers may be performing prompt eaching without an- nouncing Also, even if a provider announces prompt caching, they may not state the level of cache sharing, i.e., per-user, per-organization, or global.
However, prompt caching results in data-dependent timing
'Stanford University. Correspondence to: Chenchen Gu <cygu @esuuccs..tane cs.sstafood..duu. Tatsunori Hashimoto <thashim @ stanford.edu.>.
Proceedings of the 4rnd International Conference on Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
Therefore, we develop and conduct an audit to determine if an API provider is caching prompts and the precise level of cache sharing. Our audit uses statistical hypothesis testing and outputs valid p- values with respect to the null hypothesis
We release code and data at chenchenygu/audittnn--- pawpnngttttttaannnnnnnnnoo
com/

---

Auditing Prompt Caching in Language Model APIs
of caching, enabling guarantees on the false positive rate.
2.1. Prompt Caching
In our audit, we construct and sample response times from two procedures: one that attempts to produce cache hits, and one that produces cache misses. At high level, to attempt to produce a cache hit, we send a prompt to the API to try to cache the prompt, then we send the prompt again to try to hit the cache. To produce a cache miss, we simply send a random prompt. Under the null hypothesis of no prompt caching, where only cache misses are possible, these procedures produce identical distributions of times. Accordingly, we detect caching if we find a statistically significant difference between these distributions.
Recent works have proposed prompt caching in Transformer (Vaswani et al., 2017) LLM serving by reusing the atten- tion key-value (KV) cache across requests (Zheng et al., 2024a; Gim et al., 2024). In these methods, a prompt is cached by storing the prompt's attention KV cache. Then, if a subsequent prompt has a matching prefix with a cached prompt, the KV cache for the matching prefix can be re- trieved from the cache. As a result, cache hits will tend to have faster time to first token (TTFT), which is the time taken to process the prompt and generate the first response token. 2 In decoder-only Transformers, where each token attends only to previous tokens, reusing the KV cache for matching prefixes exactly preserves model behavior, even when the prompt suffixes differ. Figure illustrates an ex- ample of prompt caching.
We conducted audits on real-world LLM API providers in September and October 2024. We detected prompt caching in 8 out of 17 API providers. In 7 of these providers, we detected global cache sharing. On these APIs, attacker could, in principle, detect cache hits from timing differences to infer that another user sent a prompt that shares a prefix with a given prompt.
Several API providers have recently announced prompt caching features, including Anthropic (2024b), DeepSeek (2024), Fireworks (2024). and OpenAl (2024b). These providers do not state technical details of prompt caching, but these providers state that cache hits occur for (and only for) exact prefix matches between prompts. For our pur- poses, the precise implementation of prompt caching is largely unimportant. The properties of prompt caching that we exploit are:
Timing variations due to prompt eaching can also result in leakage of information about a model's architecture. Cache hits between prompts that share prefix but have different suffixes are possible only in autoregressive decoder-only Transformers, where each token attends only to previous tokens. Therefore, detecting such prompt prefix caching indicates that the model has a decoder-only architecture. Virtually all chat models are decoder-only, but embedding models can have either encoder or decoder architectures. As such, for proprietary embedding models, leakage of archi- tecture information may represent a leakage of intellectual property. By detecting prompt prefix caching, we find evi- dence that OpenAl's rext-embeddinn-3-aee --mall model has a decoder-only architecture, which was previously not pub- licly known.
1. Cache hits occur on prefix matches between prompts.
2. Cache hits tend to have a faster TTFT than cache misses (after accounting for prompt length).
To describe these properties more formally, assume that a model API takes in a prompt x and has a TTFT T(x). Note that Txx) is a random variable due to variance from network latency, server load, etc. Assume that the API has cache C, which is a set of cached prompts. If has sufficiently long matching prefix with some cached prompt e E C, then cache hit occurs.
For example, let c = "The quick brown fox jumps" and C = [c]. 1f = "The quick brown fox runs", then Tı and have a matching prefix of "The quick brown fox". so 21 could result in cache hit. On the other hand, if 2:2 = "A quick brown jumps", then 22 and c do not share prefix, so 22 results in cache miss. Since Φ and 22 are similar lengths but x1 is a cache hit and 22 is a cache miss, we would expect that TE[((::) ) < ETT())))
Responsible disclosure. In October 2024, we disclosed our audit results with each API provider in which we de- tected prompt caching. We gave providers 60 days to ad- dress the vulnerabilities before publicly releasing our find- ings, and the actual time elapsed ended up being longer. To our knowledge, at least five providers made changes to mitigate vulnerabilities, e.g.. disabling global cache sharing across organizations and updating documentation.
When a prompt & is sent to the API, we assume that x is added the cache C and that will remain in C for some fi- nite period of time. The API may use multiple servers, each with their own separate caches. We do not make assump-
2. Preliminaries and Assumptions
First, we briefly describe prompt caching, our assumptions on how users and attackers can interact with an API, and the levels of cache sharing and privacy leakage.
In emhedding models, we can view the emhedding output the first and only response "token".
2

---

Auditing Prompt Caching in Language Model APIs
Global
Per-organization caching. Each organization has its own cache, i.e., when u, who belongs organi- zation sends prompt, cache hit can occur only with a cached prompt previously sent by any user in organization o. There is a slight risk of privacy leakage certain users in the organization have access to privi- leged information that other users should not, e.g., the CEO knowing sensitive business data. However, this risk can be mitigated, the organization owner has full control over which users are members.
Organization
Organization
Usees
Figure 2.
Organizations contain users, and the global level con- tains all users and organizations of an API.
tions about how prompts are routed to servers. A prompt may be randomly routed, or may be intentionally routed to server where the prompt is already cached.
3. Global caching. The cache is shared across all users of the API, e.g., when a user sends a prompt, cache hit can occur with any cached prompt, regardless of who sent it. This leads to the highest risk of privacy leakage, as attacker could potentially learn informa- tion about any other user's prompts, including users in other organizations.
2.2. APT Assumptions
We assume that is possible to send arbitrary prompts to the API (possibly subject to some maximum length) and mea- sure the TTFT. The TTFT can be measured by setting the maximum tokens parameter to 1, which restricts the LLM output to only contain I token. Then, the overall response time is equal to the TTFT. The max tokens parameter is supported by most, if not all, woaldworld LLM APIs
3. An Audit to Detect Prompt Caching
Next, we propose an audit detect whether an API provider is caching prompts and determine the level of cache sharing. Our audit uses statistical hypothesis testing and outputs valid p-values with respect to the null hypothesis of no caching, allowing for guarantees on the false positive rate.
Fither client-side or server-side timing suffices for our pur- poses. The client-side timing is obtained simply by measur- ing the time elapsed between sending the API request and receiving the API response. The server-side timing can be measured if it is contained somewhere in the API response.
3.1. Audit Formulation: Statistical Hypothesis Testing
2.3. Levels of Cache Sharing and Privacy Leakage
To test for a given level of cache sharing, let Uvictim and Vattacker be two users that are the farthest away within that level. For example, to test for per-organization caching, Uvictim and Matnakee should be different users in the same organization.
To facilitate discussion of prompt cache sharing and privacy leakage in APIs, we define our terminology of users and organizations. A user is one person that uses the API. Each user has a unique email/username and login pass- word. An organization contains many users, but shares a billing system, centralized membership management, etc. Organizations can be used by companies, research groups, etc. Many, but not all, API providers support organizations, although sometimes under different terminology, such as teams or accounts. For consistency and simplicity, we refer to them all as organizations. Figure shows the hierarchical structure of users and organizations.
We formulate our audit statistical hypothesis test using the following null and alternative hypotheses:
Ho API does not cache prompts (at this level of sharing),
H1 API eaches prompts (at this level of sharing).
The caching in Ho does not refer only to prompt caching via the KV cache reuse described earlier. More verbosely, Hn can be written as "when Uviectim sends a prompt to the API, the API does not store any information about a that affects the TTFT T(ū) for any future prompt ă sent by Matneker" "
We consider three levels of cache sharing and their corre- sponding potential privacy leakages.
1. Per-user caching, Each user has their own cache, i.e., when user sends a prompt, a cache hit can occur only with a cached prompt previously sent by user Therefore, there is no potential privacy leakage arising from per-user prompt caching.
To test these hypotheses, we construct procedures that at- tempt produce and measure the TTFT of cache hits and cache misses. Let P be distribution of prompts. To pro- duce cache miss, Untaakkr simply sends random prompt x' ~ P to the API and measures the TTFT
'e can measure server-side timing in more than half of the APIs test, often from undocumented fields in the HTTP headers of the API response.
To attempt to produce a cache hit, first, we sample a prompt x ~ P, and Mvietim sends x to the API one or multiple times
3

---

Auditing Prompt Caching in Language Model APIs
to try to cache x. Next, we sample ā P such that ă and x share a prefix of certain length. To try produce cache hit, Mattacker sends T and measures the TTFT (T)).
Cache hit. First, we sample a random prompt or ~ P. Then, Uvictim sends x to the API NUM NUM times consecutively to try to cache z. Then, we sample I ~ P such that ō and x have shared prefix of exactly PREFIXFRACTION x PROMPTLENGTH tokens. To attempt to produce a cache hit, Uattacker sends to the API and mea- sures the TTFT T When PREFIXFRACTION = 1, we test for prompt caching when z, i.c. exact prompt matches. When PREEXFRACTION < 1, we test for prompt caching when T and x have the same prefix but different suffixes, e.g. "a b d" and "a b x".
Let Duit and miss be the distributions of TTFTss from these cache hit and cache miss procedures, respectively. Under the null hypothesis Hg of no caching, Diit- = Disss as both procedures will produce only cache misses. contrast, under the alternative hypothesis H1 of caching, we would expect the cache hit times to tend to be faster than the cache miss times, so Duit * Dmiss. Now, we can reformulate our hypotheses as
Ho: : Dnit. = Dmisss
Statistical testing. Putting these pieces together, per- form the audit, we collect NUMSAMPLES timings each from the cache hit and cache miss procedures. We randomize the order in which we collect the timing samples. Then, we test for a statistically significant difference between the distribu- tions of times from the two procedures. We use the SciPy implementation (Virtanen et al., 2020) of the two-sample Aalmogorov-Smmmoo (KS) test (Hodges Jr, 1958), which is a nonparametric test for equality of distributions. The test statistic is the maximum difference between the empir- ical cumulative distribution functions at any point. More specifically, since we expect cache hits to be faster under the alternative, we perform a one-sided test, so the test statistic is the maximum difference in the direction of cache hits being faster. The KS test outputs a p-value, which we can use to reject or not reject the null hypothesis of no prompt caching at given significance level a.
H1 : Dia + Dmiss-
Given this reformulation, to perform our audit, we first sample TTFTs from the cache hit and cache miss proce- dures. Then, we run a statistical test for whether our sam- ples came from the same distribution, c.g., the two-sample Kolmogorov-Smimot test, producing a p-value with respect to the null hypothesis of no caching.
3.2. Audit Implementation Details
Next, we describe the concrete implementation details of our audit. The procedure uses the following configuration pa- rameters: PROMPTLENGTH, PRERXFRACTION, NUM VIC- TIMREQUESTS, and NUMSAMPLES. The meanings of these parameters will become clear in the descriptions below.
Prompt distribution. Our distribution P of prompts a uniform distribution over all prompts consisting of PROMPTLENGTH English letters, lowercase and uppercase, each separated by space characters, e.g., "m Because all commonly used byte-pair encoding (BPE) tok- enizers (Gage, 1994; Sennrich et al., 2016) split on whites- pace during pre-tokenization, all prompts in P will be ex- actly PROMPTLENGTH tokens long. 5
4. Auditing Real-World APIs
Next, we audit real-world LLM APIs to identify APIs that cache prompts and determine the level of cache sharing, i.c., per-user, per-organization, or global. Cache sharing results in potential privacy leakage, as attacker could, in principle, identify cached prompts using timing data to learn information about other users' prompts.
R".
Cache miss. Uivictim sends random prompt x: P to the API and measures the TTFT T(x). Since the prompt con-
4.1. Audit Setup and Configuration
API providers and models. We audit 17 API providers: Anthropic, Amazon Bedrock, Microsoft Azure OpenAL, Co- here, Deep Infra, DeepSeck, Fireworks AI, Google, Groq, Hyperbolic, Lepton AI, Mistral, OctoAl, OpenAI, Perplex- ity, Replicate, and Together AI. The model APIs that we audit for each provider are included in Tables and 2. For API providers that primarily serve open-weight models, we audit their Llama 3 or 3.1 8B Instruct API (Dubey et al., 2024). For providers that serve proprietary models, we au- dit the cheapest chat model in their most recent family of models. In addition, we audit APIs for proprietary embed- ding models, where available. We do not audit APIs for open-weight embedding models because we did not find any
sists of random letters, there is negligible probability that a noticeable prefix has already been cached: the probability that two prompts sampled from P share a prefix of 15 to- kens or longer is less than 10 25_ Therefore, this procedure accurately measures a distribution of cache miss times.
"Multiple requests may be necessary in some scenarios, e.g., if API requests are randomly routed to one of several servers, and each server has separate cache.
* "Many APIs add a small number of tokens to the user prompt due to the default system prompt, special tokens for prompt and role formatting, etc. However, these additional tokens are unim- portant for our procedure, as the number of additional tokens is small and remains constant across prompts to a given model API.
4

---

Auditing Prompt Caching in Language Model APIs
Table 1. Audit results for APIs where detected prompt caching. / denotes caching detected, X denotes caching not detected, and denotes that cache sharing within an organization was not tested, either because the API did not support organizations or because did not have access to the organizations feature. We report the average precision for classifying times from the cache hit procedure, using the highest level of cache sharing detected in each API. We report the average precision for client-side timing and server-side timing separately, with "_" denoting that the given timing method unavailable for that API.
Same prompt
Same prefix, different suffixes
Avg. precision
Provider
Model
Per-user
Per-user
Per-org.
Global
Client
Server
Azure
(ext-embedding-3-e
0.80
Deep Infra
Llama 3.1 8B Instruct
0.84
Fireworks
Llama 3.1 8B Instruct
0.77
0.79
Lepton
Llama 3.1 8B Instruct
0.71
0.70
OpenAl
(ext-embedding-3-smil
0.78
0.79
Perplexity
Llama 3.1 8B Instruct
0.90
Replicate
Llama 3 8B Instruct
1.00
Anthropic
Claude 3 Haiku
X 0.84
OpenAl
GPT-4o mini
X
0.79
0.86
Table Audit results for APIs where did not detect prompt caching. X denotes that caching was not detected.
by setting PREFIXFRACTION 1. We set Ulyictim and Matnakker to be the same user, and we set NUNVICTIMREQUESTS = 25.
Same prompt Per-user
Provider
Model
In the remaining three levels, we test for prompt caching when 0 and have the same prefix but different suffixes by setting PREMXFRACTION
Amazon
Claude Haiku
Azure
GPT-4o mini
X
0.95. We test for increasing levels of cache sharing by appropriately setting the victim and attacker users:
Cohere
Command R
X
Cohere
embed-english-v3..0
DeepSeek
DeepSeek Chat
Google
Gemini 1.5 Flash
Per-user caching, Uvictim and Mattacker are the same user, in the first level.
Google
eext-emeedding-O0
Groq
Llama 8B Instruct
Hyperbolic
Llama 3.1 8B Instruct
3. Per-organization caching, tiicctm and Mattacker are dif- ferent users within the same organization. For APIs without organizations, we skip this level.
Mistral
Mistral Nemo
X
Mistral
Mistral Embed
OctoAI
Llama 3.1 8B Instruct
Together
Llama 3.1 8B Instruct
4. Global caching, Uvvccti and Uattackrr are different users different organizations. For APIs without organiza- tions, Uvictim and Uattacker are simply different users.
APIs that served open-weight decoder-only Transformer em- bedding models. Prefix caching is possible in decoder-only Transformers but not encoder-only Transformers, where each token attends all other tokens in the prompt.
In levels 2-4, to determine how many victim re- quests are needed to detect caching, we run tests using NUMVICTIMREQUESTS
{1, 5, 25} in increasing order, stopping after the first significant p-value. To account for multiple testing, we perform Bonferroni correction by dividing the significance threshold for each test by three.
Configuration and procedure. For our audits, we use PROMPTLENGTH 5000 and NUMSAMPLES = 250. We run four levels of audits of increasing cache sharing and privacy leakage. At each level, we only continue to audit APIs we detect caching during the previous level. We use a significance level of
In all levels, if only one timing method is available in an API (client-side or server-side timing), then we use that timing method. If both are available, we run tests using both timing methods and perform another Bonferroni correction, dividing by two this time.
To narrow down our list of providers, the first level tests for the simplest level of prompt caching:
1. Same prompt, per-user caching. We test for prompt caching on exact prompt matches (i = r)
Cost per test. When NUM NUMVICTIMREQQESTTS = 25, one test uses roughly 34 million prompt tokens. The number of
5

---

Auditing Prompt Caching in Language Model APIs
Anthropic Claude-3 Halku
Azure embbddding plobal, client time, v = 25
Deep Infra Liama 3.1 88 global, client time, v =
Fireworks Liama 3.1 88
per.arg.. client time, V
g @jobal, client time, V =
Cache miss
Time
Firwwoks Liama 8B
Lepton Liama 3.1 BB global, client timee, V
OpenAi GPT-4o mini poriarg.. client time, v
OpenAi GPT-4o mini
global, server time, v
per-arg, server time,
hequancy
0.0
3.44
0.55
3:55
8,6
0.30
3.9
131
92
2.4
0.
1.0
92
03
0.4
0.5
8.6
677
3.1
0.9
0.50
015
333
622
0.30
335
C/C
0/5
15
OpenAl 3mbedding-3-smalll
OpenAl embedding 3-ssmaall
Perpiexity Liama 3.1
Replicate Liama 3
global, client
25
global, time, 25
global, client time,
global, time,
3.5
0.20
0.35
1.50
3.35
8.48
0.35
0.50
0.15
3.23
9.5
0.50
8.25
0.35
0.00
9:45
6.24
05
[n]
T0me 30
Time (1)
Time (4)
Figure 3. Histograms of response times from the cache hit and cache miss procedures in APIs where we detected caching. The distributions of times are clearly distinguishable, with eache hits tending to be faster. Each histogram title states the API provider, model, level of cache sharing (per-org. or global). timing source (client-side or server-side timing), and the NUNVICTIMREQQUSTS used, denoted V.
response tokens used is much smaller because we set the maximum response tokens parameter to 1. For the chat APIs we audit, the prices per million prompt tokens are 0.05-0.25 USD, resulting in a cost per test of 1.69-8.4 44 USD. The tests are cheaper when NUMVICCTIMRQQUESST smaller.
Figure 3 displays histograms of times from the cache hit and cache miss procedures. The distributions of times are clearly distinguishable, with cache hits tending to be faster. Each histogram title states the minimum NUMVICTIMRE QUESTS (denoted in the titles) that resulted in a significant p-value. In most of the APIs where we detected caching, only NUNVICTIMREQQESTE
4.2. Audit Results
was needed to detect caching. Only the OpenAl and Azure text-embedding- 3-small APIs required SUMVICTIMREQQUSTS = 25 to achieve a significant p-value. This may suggest that these APIs have multiple servers with separate caches and that requests are randomly routed to a server, so multiple victim requests are needed cache the prompt in enough servers for the attacker's ' prompt to have sufficient probability of producing a cache hit. In Appendix B, we report all the p-values from our audits. In many APIs, the p-values are many orders of magnitude smaller than our significance level of a = 10 8 In all APIs where we detected caching, all available timing methods resulted in significant p-values.
We conducted our audits in September and early October 2024 using clients located in California. Table 1 shows audit results for APIs in which we detected prompt caching, and Table 2 shows APIs in which we did not detect prompt caching. We detected prompt caching in 8 out of 17 API providers, and we detected global cache sharing in 7 providers. This means that an attacker can potentially learn information about other users' prompts by identifying cached prompts from timing data. To assess an attacker's ability to distinguish between cache hits and cache misses, Figure 4 contains selected precision recall curves for clas- sifying times from the cache hit procedure. 6 The curves show that cache hits can be detected with near perfect preci- sion up to moderate recall scores. Figure 6 in the appendix shows precision-recall curves for the highest level of cache sharing we detected in each API. To numerically summarize these curves, we compute the average precision (Zhu, 2004), which is equal to the area under the precision-recall curve (the precision is averaged over the interval of recall scores from 0 to 1). Table 1 shows that the average precisions mostly lie around a moderately high value of 0.8.
In the Anthooic Claude 3 Haiku and OpenAl GPT-4o mini APIs, we detected per-organization cache sharing, but not global cache sharing. This exact level of cache sharing is stated in their prompt caching documentations, confirm- ing the efficacy of our audit procedure. Since OpenAl (2024a) and Anthropic (2024a) document per-organization cache sharing, we do not consider a security vulnerability. Global cache sharing in the OpenAl text-embedding-3 3-small API was a potential vulnerability, but has been patched after our responsible disclosure prior to the release of this paper.
'The cache hit procedure attempts to produce cache hits but cannot guarantee eache hits (e.g.. due to server routing), so some times the cache hit distribution may actually be cache misses.
Although DeepSeek (2024) has a prompt caching feature and returns the number of cache hit tokens in API responses,
6

---

Auditing Prompt Caching in Language Model APIs
OpenAl GPT-4o mini per-org. server time,
Perplexity Llama 3.1 8B global, client time,
Relationship to p-values. Figure in Appendix C shows the effects of the ablations the audit p-values. We ob- serve similar patterns as above, with decreases in average precision corresponding to increases in p-values.
0.8
0.8
Preciston
0.6-
Precision
0.6
4.4. Difficulty of Prompt Extraction Attacks
Our results show that given specific prompt I, an attacker could potentially detect cache hits to learn whether another user sent a prompt that shares a prefix with x. A natural question is whether an attacker could extract other users' prompts token-by-token. One idea is to use breadth- search: given a partial candidate prompt, such an attack would try possible continuation tokens and determine which continuation token is cached. The cached token is appended to the candidate prompt and the process repeats.
0.2
Average precision i 0.06
Average preciston 0,90
0.0
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.8
0.8
1.0
Recall
Recall
Figure 4. Selected precision-recall curves for distinguishing be- tween times from the cache hit and cache miss procedures. Cache hits are the positive class. The curves show that cache hits can be detected with near perfect precision up to moderate recall scores. Figure 6 in the appendix contains curves for other APIs.
which we used to confirm that we produced cache hits, we were unable to detect caching from response times. There was no statistically significant difference between the distri- butions of cache hit and cache miss times, even in two-sided tests. DeepSeck states that the cache is isolated per-user, and we empirically verified that this is the case based on the number of cache hit tokens returned in the API responses.
However, we were unable to execute practical prompt extrac- tion attacks. A successful attack requires extremely accurate detection of cached tokens, as there are many possible con- tinuation tokens at each step. Just one incorrect token causes complete failure due the exact prefix match required for cache hit. In preliminary experiments, we were unable to reliably detect the presence of one additional cached token. It is also difficult to make repeated measurements to boost accuracy. To detect whether prompt is cached, the attacker must send the prompt to the API. Then, future measure- ments may produce a cache hit not because another user sent the prompt, but because the attacker sent it.
4.3. A Ablations
We run ablations to determine the effects of PROMPTLENGTH. PREFXXFRACTION, and model size the average precision, shown in Figure 5. We use the APIs in which we detected global caching with NUMVICTIMREQUESTTS = 1, i.e., the Llama 3 or 3.1 8B Instruct APIs of Fireworks, Perplexity, and Replicate.
We do not claim that prompt extraction attacks are necessar- ily impossible. Such attacks face difficulties, but future work may yet develop successful, practical attacks. In addition, in more restricted sets of target prompts, c.g., known prompt templates with places for users to enter private personal information, may be easier to overcome these difficulties.
Smaller PROMPTLENGTH decreases average precision. In Figures 5a and 5b, we vary the PROMPTLENGTH in the 1) and same prefix
same prompt (PREFXFRACTION but different suffixes (PREAXFRACTION = 0.95) settings, respectively. When the PROMPTLENGTH is moderately high &2 1000). the average precision is relatively high and stable. However, as the PROMPTLENGTH approaches zero, the average precision decreases to random chance.
5. Leakage of Architecture Information
In addition to privacy implications, the detection of prompt caching can also reveal information about a model's 's archi- tecture. This is because the conditions for cache hits to occur depend on model architecture.
Decreasing PREFIXFRACTION decreases average preci- sion. In Figure 5c, we vary the PREFIXFRACTION while setting PROMPTLENGTH = 1000. As the length of the matching prefix decreases, the average precision decreases to random chance.
In decoder-only Transformer models, reuse of the attention KV cache enables cache hits between prompts with match- ing prefixes, even if the suffixes differ, since each token attends only to previous tokens. This prefix caching is not possible in encoder-only or mncoder-decoder Transformer models, where each token in the prompt attends to all other tokens in the prompt. Therefore, detecting such prompt prefix caching indicates that model cannot have bidirec- tional encoder architecture. Virtually all chat models are decoder-only, but embedding models can have either en- coder or decoder architectures, as seen in the Massive Text Embedding Benchmark (MTEB) leaderboard (Muennighoff
No clear relationship between model size and average precision. In Figure 5d, we vary the model size on the Fireworks API, which supports all models in the Llama 3.1 and 3.2 families. We detected caching in all model sizes, with no clear relationship between model size and average precision.
7

---

Auditing Prompt Caching in Language Model APIs
(x) Vary prompt length Isame prompt)
(b) Vary prompt length (same prefix, diff, suffix) 3.0-
(c) Vary prefix match length
(d) Vary model size
1.0
a
10-
x
Merrgg precsiin
Average precision
Avaroge prectsion
0.9
Nverggg presision
3.9
0.8
2.8
0.8
0.5
0.7
0.7
0.7
0.7
0.6-
1000
2000
3003
4000
5000
1000
2000
3000
4000
5000
200
600
1000
1h
110
750
4058
Prompt length (takens)
length Itakens)
Prefix match length (tokens)
Liamaa model sizee
Fireworks Iclient time)
Fireworkss (server time)
Perplexity (client time)
Replicate (server time)
Random
Figure 5. Ablations on the effects of PROMPTLENGTH, PREFIXFRACTION, and model size on the average precision. In (a)-(c), as the prompt length or prefix match length decreases, the average precision decreases random chance. In (d), we detect caching across all model sizes, with no clear relationship between model size and average precision.
caching, an attacker will not be able to produce cache hits et al., 2023). As such, for proprietary embedding models, leakage of architecture information may represent a leakage of intellectual property.
prompts sent by other users. Since it is unlikely that dif- ferent users will send prompts with long matching prefixes, per-user caching should retain many of the performance benefits from global cache sharing.
In our audits (Table 1), we detected prompt caching in OpenAl's lext-embedding---smaal API when prompts had the same prefix but different suffixes. We confirm that when the prompt suffix is changed, the returned embed- ding also changes, indicating that the caching mechanism does not simply return cached embedding outputs from similar prompts. Assuming that -et--embedding---smmaee is Transformer-based, this indicates that text-embedding-3 small a decoder-only Transformer. This is new informa- tion, as OpenAl has not released any information about the architecture of their embedding models.
Disclosure of cache sharing. We believe that providers should disclose their caching policies, particularly the level of cache sharing. It is important that users know how their data is handled and who could potentially learn information about their data. This way, users can make informed deci- sions about how they use an LLM API. For example, if a company knows that an API uses per-organization cache sharing, the company can decide to create separate organi- zations for different groups of employees to prevent umau- thorized information access.
Floating-poin precision of the cache. When we send the exact same prompt multiple times, when the response time is noticeably faster, indicating a cache hit, the returned embed- ding differs slightly from the "normal" embedding in most of the responses with normal response times, which indicate cache misses. This behavior is consistent across different random prompts. These differences are small, on the order of 10-4 to 10-5 in each coordinate. We hypothesize that these differences may arise if the reused KV cache is stored in a lower floating-poit precision, resulting in slight dis- crepancies when the attention KV is computed from scratch in cache misses versus when it is retrieved from the cache in cache hits. Interestingly, in some responses, especially those that are noticeably slower, the embedding differs from both the "normal" and "cache hit" embeddings. This may be caused by some responses being processed by different GPU models, as floating point computations can differ slightly across different GPUs. Appendix D contains examples of response times and embeddings showing this phenomenon.
Disabling caching prevents any information leakage. For information leakage that only requires per-user eaching, such as leakage of architecture information, the strongest mitigation is to disable prompt caching. Since per-user caching does not result in privacy leakage, but may result in leakage of the API provider's ' intellectual property. it is up to the provider to determine their level of risk tolerance. Another potential mitigation is to intentionally delay the re- sponse time for cache hits so that they look like cache misses. This eliminates the benefits of prompt caching for users, but API providers could still benefit, cached prompts require less GPU processing time.
7. Related Work
Prompt caching. Many recent works have developed opti- mizations for inference and serving of Transformer language models. Various methods involve reuse of the attention KV cache, improving latency and throughput for shared prompt prefixes (K woo et al., 2023; Zheng et al., 2024a; Gim et al., 2024; Ye et al., 2024a;b; Qin et al., 2024: Juravsky et al., 2024). Recall that we do not assume any particular imple- mentation of prompt caching in our attacks. Indeed, we do not know technical details about the eaching mechanisms
6. Mitigations
Per-user caching prevents privacy leakage. To com- pletely prevent any privacy leakage from prompt caching, only per-user caching should be allowed. per-user
8

---

Auditing Prompt Caching in Language Model APIs
used by the APIs we audited. Other caching methods do not preserve exact model behavior, such as retrieving cached responses for semantically similar prompts (Bang, 2023) or reusing the KV cache even when the prefixes do not exactly match (Gim et al., 2024; Yao et al., 2025; Hu et al., 2024). We do not study such methods, but they are also likely sus- ceptible to similar cache timing attacks, and our audit can easily be adapted to detect other types of caching.
8. Conclusion
As LLMs and other machine learning systems become more widely deployed and used in the real world, it is increasingly important consider security and privacy aspects of these systems. To this end, in this paper, we find that prompt caching in LLM APIs can leak private and proprietary infor- mation through timing differences. We develop and conduct rigorous statistical audits on real-world APIS, finding that multiple APIs were performing global cache sharing. We hope that future work will continue to evaluate and audit the security and privacy of machine learning systems, ensuring their robustness and trustworthiness.
Cache timing attacks. In computer security, many side- channel timing attacks have extracted information by using timing differences to distinguish between cache hits and cache misses, e.g., in the CPU cache web cache. For example, cache timing attacks have been used to extract AES keys (Bernstein, 2005; Osvik et al., 2006; Bonneau & Mironov, 2006; Tromer et al., 2010; Gullasch et al., 201 1; Yarom et al., 2017), a user's private web information (Felten & Schneider, 2000; Bortz& & Boneh, 2007; Van Goethem et al., 2015), and sensitive data from other processes on machine (Percival, 2005; Yarom& & Falkner, 2014; Liu et al., 2015), as in the well-known Meltdown (Lipp et al., 2018) and Spectre attacks (Kocher et al., 2019).
Impact Statement
Responsible disclosure. As discussed earlier, to mitigate real-world harms arising from our research, we performed responsible disclosure. In October 2024, we disclosed our audit results with each API provider in which we detected prompt caching. We gave providers 60 days to address the vulnerabilities before publicly releasing our findings, and the actual time elapsed ended up being longer. To our knowledge, at least five providers made changes to mitigate vulnerabilities, e.g., disabling global cache sharing across organizations and updating documentation.
Attacks language model APIs. Several recent works have attacked language model APIs. Carlini et al. (2024) and Finlayson et al. (2024) show that logits and logprobs leak in- formation from an LLM API, including the model's s hidden dimension size and final layer weights. Weiss et al. (2024) partially extract encrypted and streamed LLM responses by inferring and analyzing token lengths from packet sizes. Carlini & Nasr (2024) and Wei et al. (2024) exploit specu- lative decoding (Leviathan et al., 2023; Chen et al., 2023) and similar methods to extract LLM responses with higher success by measuring delays between packets.
Broader impact. We believe that our audits for detecting prompt caching and the level of cache sharing in LLM APIs can improve transparency and trust. By increasing transparency around eaching policies and how user data handled, users can make better informed decisions about how they use an LLM API and have the appropriate level of trust that their data will be secure and private. More broadly, we believe that audits are a promising method to ensure that machine learning systems are safe, secure, and trustworthy, especially as these systems become more widely deployed and have larger socictal impact.
Most related to our work are Song et al. (2024) and Zheng et al. (2024b), which also study timing attacks and privacy leakages arising from prompt caching, including both KV cache reuse and semantic caching, primarily in simulated, controlled environments. Our work differs in developing an audit that is practical and provides statistical guarantees, using these audits to precisely identify different levels of cache sharing, and extracting information about model ar- chitecture. Song et al. (2024) demonstrate prompt extraction attacks in a simulated setting, but the attack is run locally without network latency, uses knowledge of the distribution of prompts, requires explicit clearing of the cache to make repeated measurements, and makes an average of over 200 measurements for each extracted token. Due to these limita- tions, we believe that these simulated attacks are currently unlikely to be real-world privacy threats.
A Acknowledgments
CG was supported by the Stanford CURIS program. XL was supported by a Two Sigma PhD Fellowship. PL was supported by NSF Award Grant no. 1805310 and an Open Philanthropy Project Award. TH was supported by gifts from Open Philanthropy, Amazon, Google, Meta, and a grant under the NSF CAREER IIS-2338866.
Conflicts of Interest
PL is a co-founder of Together AI. However, this work was done in his Stanford capacity. The methods, providers audited, and results were not influenced by or shared with Together prior to the public release of this paper. All API
9

---

Auditing Prompt Caching in Language Model APIs
providers were audited using the same procedure, including Together. When this work conducted, none of the other authors had conflicts of interest with the providers audited in this paper.
Chen, C., Borgeaud, S., Irving, G., Lespiau, J.-B., Sifre, L., and Jumper, J. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01318, 2023.
DeepSeek.
Deepseek api introduces context caching on disk, cutting prices by an order of magninude. https: /web. / wwt.a/ee..rrr//eee/2222222000000
References
Anthropic. Prompt caching (beta). https:
h/w.bbarchhia archivee hwwp.//errggeee//222222222222220
/tttss://ocs. docs. wwwhropii.coooo/aaaaaaaa cttp:///cccccco/////oc///ooo/sss
2024.
2024a.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan,
Anthropic. Prompt caching with claude. https:
A., et al. The llama 3 herd of models. arXiv preprint 2024.
https: http::///www.entrropicc.ooo/o///
prompt-caching 2024b.
Felten, E. W. and Schneider, M. A. Timing attacks on web privacy. In Proceedings of the 7th ACM Conference on Computer and Communications Security, pp. 25-32, 2000.
Bang, F. GPTCache: An open-source semantic cache for LLM applications enabling faster answers and cost savings. Tan, L.,, Milajevs, D., Chauhan, G., Gwin- nup, J., and Rippeth, E. (eds.), Proceedings of the 3rd Workshop for Natural Language Processing Open Source Software (NLP-OSS 2023), pp. 212-218, Singapore, De- cember 2023. Association for Computational Linguis- tics. doi: 10.186633w1/222231pposs 1.24. URL https: //aclanthology.oog http://2222.....me11.... n1poss-1.24...
Finlayson, M., Ren, X., and Swayamdipta, Logits of API- protected LLMs leak proprietary information. In First Conference on Language Modeling, 2024. URL https: / //openreview net/ httt://cruumm.cccoccmmmmp..
Fireworks. Prompt caching, https://web.archive. org/ /tw////bb/beeee / wwtpe//00001100333attttpoo//a//0
Bernstein, D. J. Cache-timing attacks on aes. 2005. URL https://cet .p.rry..tee wtt.tt//foggoggr/////. cachet
/ 2024.
Gage, P. A new algorithm for data compression. The Users Journal, 12(2):23-38, 1994.
Bonneau, J. and Mironov, I. Cache-collision timing attacks against aes. In Cryptographic Hardware and Embedded Systems-CHES 2006: 8th International Workshop, Yoko- hama, Japan, October 10-13, 2006. Proceedings 8. pp.
Gim, I., Chen, G., Lee, S.-s., Sarda, N., Khandelwal, A., and Zhong, L. Prompt cache: Modular attention reuse for low-latency inference. Proceedings of Machine Learning and Systems, 6:325-338, 2024.
201-215. Springer, 2006.
Bortz, A. and Boneh, D. Exposing private information by timing web applications. In Proceedings of the 16th international conference on World Wide Web, pp. 621 628. 2007.
Gullasch, D., Bangerter, E.,, and Krenn, S. Cache games- bringing access-based cache attacks on aes to practice. In 201 IEEE Symposiun on Security and Privacy, pp. 490-505 IEEE, 2011.
Carlini, N. and Nasr, M. Remote timing attacks on efficient language model inference. arXiv preprint arXiv:2410.1711 17175, 2024.
Hodges Jr, J. The significance probability of the smirnov two-sample test. Arkiv för matematik, 3(5):499-44) 486, 1958.
Carlini, N., Paleka, D., Dvijotham, K. D., Steinke, T., Hayase, J., Cooper, A. F., Lee, K., Jagielski, M., Nasr, M., Conmy, A., Wallace, E., Rolnick, D., and Tramèr, F. Stealing part of production language model. In Salakhutdinov, R., Kolter, Z., Heller, K., Weller, A., Oliver, N., Scarlett, J., and Berkenkamp, F. (eds.), Pro- ceedings of the 41st International Conference on Ma- chine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 5680-5700 5705. PMLR, 21-27 Jul 2024. URL BtRhhtps:: http:::://pppccpepeeeeee mlr. press/ 24a html.
Hu, J., Huang, W., Wang, H., Wang, W., Hu, T., Zhang, Q. Feng, H., Chen, X., Shan, Y., and Xie, T. Epic: El- ficient position- innttinnnideeeedeent context caching for serving large language models. arXiv preprint arXiv:2410. 15332,, 2024.
Juravsky, J., Brown, B., Ehrlich, R., Fu, D. Ré, C., and Mirhoseini, A. Hydragen: High-throughput Ilm inference with shared prefixes. arXiv preprint arXiv:2402.00 000999 2024.
10

---

Auditing Prompt Caching in Language Model APIs
Kocher, P., Horn, J., Fogh, A., Genkin, D., Gruss, D., Haas, W., Hamburg, M., Lipp, M., Mangard, S., Prescher, T., Schwarz, M., and Yarom, Y. Spectre attacks: Exploit- ing speculative execution. In 2019 IEEE Symposium on Security and Privacy (SP), pp. 1-19, 2019. doi: 10.1109/SP2211.000000
Conference 2006, San Jose, CA, USA, February 13-17, 2005. Proceedings, pp. 1-20. Springer, 2006.
Pereival. C. Cache missing for fun and profit. BSDCan Ottawa, 2005. URL https: http://wwwwa/mma daemenology. net/papers/ htttts//ttttttttte htt.//tt........
Qin, R., Li, Z., He, W., Zhang, M., Wu, Y., Zheng, W., and Xu, X. Mooncake: A kvcache-centric disag- gregated architecture for Ilm serving. arXiv preprint arXiv:2407.00079 2024.
Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J., Zhang, H., and Stoica, I. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Sym- posium on Operating Systems Principles, pp. 611-626, 2023.
Sennrich, R., Haddow, B., and Birch, A. Neural machine translation rare words with subword units. In Erk, K. and Smith, N. A. (eds.), Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1715-1725, Berlin, Germany, August 2016. Association for Compu- tational Linguistics. doi: 10. 10.18653//1/P16--1166 URL https: / haa/aaalaaanhlleeeeeae
Leviathan, Y.. Kalman, M., and Matias, Y. Fast in- ference from transformers via speculative decoding. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.), Proceedings of the 40th International Conference on Machine Learn- ing, volume 202 of Proceedings of Machine Learn- ing Research, pp. 19274-19286. PMLR, 23-29 Jul 2023. httptss:: / http://pcoceeee...eeeeee/o .ml..mmrr... press/ 22.2/leviathan222 htmml.
Song, L.,, Pang. Z., Wang, W., Wang, Z., Wang. X., Chen, H., Song, W., Jin, Y., Meng, D., and Hou, R. The early bird catches the leak: Unveiling timing side channels in Ilm serving systems. arXiv preprint arXiv:2409.20002. 2024.
Lipp, M., Schwarz, M., Gruss, D., Prescher, T., Haas, W., Fogh, A., Horn, J., Mangard, S., Kocher, P., Genkin, D., Yarom, Y., and Hamburg, M. Meltdown: Reading kernel memory from user space. In 27th USENIX Security Symposium (USENIX Security 18), 2018.
Tromer, E., Osvik, D. A., and Shamir, A. Efficient cache attacks aes, and countermeasures. Journal of Cryptol- ogy, 23:37-71, 2010.
Liu, F., Yarom, Y., Ge. Q.. Heiser, G., and Lee, R. B. Last- level cache side-channel attacks are practical. In 2015 IEEE symposium on security and privacy, pp. 605-622. IEEE, 2015.
Van Goethem, T., Joosen, W., and Nikiforakis. N. The clock is still ticking: Timing attacks in the modern web. In Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security, pp. 1382-1393, 2015.
Muennighoff, N., Tazi, N., Magne, L., and Reimers, N. MTEB: Massive text embedding benchmark. In Vla- chos, A. and Augenstein, I. (eds.), Proceedings of the 17th Conference of the European Chapter of the Asso- ciation for Computational Linguistics, pp. 2014-2037, Dubrovník, Croatia, May 2023. Association for Compu- tational Linguisties. doi: 10. 10.18653/1/20033eelllmmi 148. URL https:://///...... http://aaaaatthhn http:////caaaaaallgggooo//////// eacl wwi..ann......... 148/.
Vaswani. A., Shazeer, N., Parmar, N., Uszkoreit. J., Jones, L., Gomez, A. N., Kaiser, L. u., and Polosukhin, I. Attention is all you need. In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https:: hetp::////ppoeeennnngeeerrr... neuripp... cc cc/paper oc/sappe_tiiee/pppe///////LL 3 3faa22245440ee991110053333444643 pdf.
OpenAL. Prompt caching. https://platfore
openai cww.cocs/ guides/i///ggg 2024a.
Virtanen, P., Gommers, R., Oliphant, T. E., Haberland, M., Reddy, T., Cournapeau, D., Burovski, E., Peterson, P., Weckesser, W., Bright, J., van der Walt, S. J., Brett, M.. Wilson, J., Millman, K. J., Mayorov, N., Nelson, A. R. J., Jones, E.,, Kern, R., Larson, E., Carey, C. J., Polat, i., Feng, Y., Moore, E. W., VanderPlas, J., Laxalde, D., Perktold, J., Cimrman, R., Henriksen, I., Quintero, E. A., Harris, C. R., Archibald, A. M., Ribeiro, A. H., Pedregosa, E,, van Mulbregt, P., and SeiPy 1.0 Contributors. SciPy
OpenAL. Prompt
caching in the api.
btttps:/web.ardchee 2044103395507//ttppss///o index/api wwtppppooottccciiie 2024b.
com/
Osvik, D. A., Shamir, A., and Tromer, E. Cache attacks and countermeasures: the case of aes. In Topics in Cryptology- CT-RSA 2006: The Cryptographers' Track at the RSA

---

Auditing Prompt Caching in Language Model APIs
1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods, 17:261-272, 2020. doi: 10. 00.1088441592--01--066--
Systems, 2024a. URL https: / www.//veen.eeweeeeeoeeee net/ Forum??d=vVaaRkobbbg
Zheng. X., Han, H., Shi, S., Fang. Q. Du. Z., Guo, Q. and Hu, X. Inputsnatch: Stealing input Ilm ser- vices via timing side-channel attacks. arXiv preprint urXiv:241..18199 2024b.
Wei, J., Abdulrazzag, A., Zhang, T., Muursepp, A., and Saileshwar, G. Privacy risks of speculative decoding in large language models. arXiv preprint 2xXiv:241..01077 2024.
Zhu, M. Recall, precision and average precision. Depart- ment of Statistics and Actuarial Science, University of Waterloo, Waterloo, 2(30):6, 2004.
Weiss, R., Ayzenshteyn, D., and Mirsky, Y. What was your prompt? a remote keylogging attack on Al assis- tants. In 33rd USENIX Security Symposium (USENIX Security 24), pp. 3367-3384, Philadelphia, PA, August 2024. USENIX Association. ISBN 978-1-939133-444 URL https::/// //wwtt www.::.www......oooooooooooooooe usen ixeenxseee www.xxeeecxtt... bo
Yao, J., Li, H., Liu, Y., Ray, S., Cheng, Y., Zhang, Q., Du, K., Lu, S., and Jiang, J. Cacheblend: Fast large language model serving for rag with cached knowl- edge fusion. In Proceedings of the Twentieth Euro- pean Conference on Computer Systems, EuroSys '25, pp. 94-109, New York, NY, USA, 2025. Association for Computing Machinery. ISBN 7798400711961 doi: 10. 1145/3689031 13696998. URL https::pp//.. org/
10. N0.1145/3669031.. 3696098.
Yarom, and Falkner, K. (FLUSH+ RELOAD): A high resolution, low noise, 13 cache ChdenChhnneel) attack. In 23rd USENIX security symposium (USENIX security 14), pp. 719-732, 2014.
Yarom, Y., Genkin, D., and Heninger, N. Cachebleed: a timing attack on openssl constant-time rsa. Journal of Cryptographic Engineering, 7:99-112, 112, 2017.
Ye, L., Tao, Z., Huang, Y., and Li, Y. ChunkAttention: Efficient self-attention with prefix-aware KV cache and two-phase partition. In Ku, I..W., Martins, A., and Sriku- mar, V. (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Pupers), pp- 11608-11620, Bangkok, Thailand, August 2024a. Association for Computational Linguis- tics. doi: 10. 10.:6655w.11222244cc4ooogg..... //aclaanthooog..orggg.ooo. hww.hhooogy...ogg/////eo
URL https: aon.anallln..n..c.....emcee 623.
Ye. Z.. Lai, R., Lu, B.-R., Lin, C.-Y., Zheng, S., Chen, L., Chen, T., and Ceze, L. Cascade inference: Mem- ory bandwidth efficient shared prefix batch decoding, February 2024b. URL /tttps://llashinfft ai/ 2024 /0224/02/00 cwscade- insscade-ifference html.
Zheng, L.,, Yin, L., Xie, Z., Sun, C.,, Huang, J., Yu, C. H., Cao, S., Kozyrakis, C., Stoica, I., Gonzalez, J. E., Barrett, C.. Sheng. Y. SGLang: Efficient execution of struc- tured language model programs. In The Tiihth-eighth Annual Conference on Neural Information Processing
12

---

Auditing Prompt Caching in Language Model APIs
A. Precision-Recall Curves
Figure shows precision-recall curves for distinguishing between cache hit and cache miss times in APIs where we detected caching our audits (Table 1).
B. P-values from Audits
We report all the p-values from our audits on APIs. Table contains p-values from level of our audits: same prompt, per-user caching. Table 4 contains p-values from level 2 of our audits: prompts with the same prefix but different suffixes, per-user caching. Table contains p-values from level 3 of our audits: prompts with the same prefix but different suffixes, per-organization caching. Table contains p-values from level 4 of our audits: prompts with the same prefix but different suffixes, global caching.
C. Ablation Effects on Audit p-values
Figure 7 shows the effects of the ablations in $4.3 on the audit p-values. Each test is run using NUMSAMPLES = 250. We observe similar patterns as in $4.3, with decreases in average precision corresponding to increases in p-values. As the prompt length or prefix match length decreases, the p-values grow larger. We detect caching across all model sizes, with no clear relationship between model size and p-values.
D. Embeddings and Response Times from the OpenAl text-embedding-3-saalll API
Tables 7, 8, and contain real examples of response times and embeddings illustrating the phenomenon discussed in $5. Each table contains the server-side response times and first five embedding coordinates when sending the same prompt 25 consecutive times to the OpenAl rext-embedding---malll API from the same user. More specifically, each table shows the victim requests for one prompt in the main audits, which used PROMPTLENGTH 5000.
The tables show that there is "normal" embedding (shown in blue) that is returned in most the responses with normal response times, which indicate cache misses. When the response time is noticeably faster (shown in green), indicating cache hit, the embedding differs slightly from the "normal" embedding. Most, but not always all, fast responses have the same embedding. In some responses, especially those that are noticeably slower, the embedding (shown in red) differs from both the "normal" and "cache hit" embeddings. There are several different "alternate" embeddings among these responses.
13

---

Auditing Prompt Caching in Language Model APIs
Anthropic 3 Haiku per-org.. client time,
Azure embedding- global, client time, 25
Deep Infra Llama 3.1 8B global, client time, v =
Fireworks Liama 3.1 8B
globai, client time,
1.d
1.0
1.0
1.0
0.8
0.8
0.8
Precision
0.6
Precision
0.6
Precision
0.6
Precision
0.6
0.4
0.2
0.2
0.2
0.2
Average precision 0.84
Average precision 0.80
precision 0.84
precision 0.77
0.0
0.0
0.6
0.2
0.6
0.8
1.0
0.2
0.6
0.8
0.6
0.8
Recall
Recall
Recall
Recall
Fireworks Llama 3.1 8B
Lepton Liama 3.1 8B
Lepton Liama 3.1
OpenAl GPT-40 mini
global, server time, v =1
global, client time, V = 5
global, server time,
per-org., client time, V =
1.0
1.0
1.0
1.0
Precisian
0.6
Precision
0.6
Precision
0.6
Precision
0.6
0.4
0.4
0.2
0.2
0.2
0.2
Average precision 0.79
Average precision - 0.71
Average precision 0.70
Average precision 0.79
0.0
0.0
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Recall
OpenAl GPT-4o mini
OpenAl embedding 3-small global, client time, = 25
OpenAl embedding-3 3-mlll global, server time, v = 25
Perplexity Liama 3.1 8B
per-org., server time, V
global, client time, v 1
1.0
1.0
1.0
0.8
0.8
0.8
Precisian
0.6
Procision
0.6
Precision
0.6
Precision
0.6
0.4
0.4
0.4
0.4
0.2
02
0.2
Average precision - 0.86
Average precision - 0.78
Average precision - 0.79
Average precision 0.90
0.0
0.0
0.0
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Recall
Recall
Recall
Replicate Llama 8B
global, server time, v
1.0
Average precision
Precision
0.4
0.0
0.0
0.2
0.4
0.6 Recall
0.8
1.0
Figure 6. Precision-recall eurves for distinguishing between times produced by the cache hit and eache miss procedures in APIs where we detected caching in our audits (Table 1). Cache hits are the positive class, and cache misses are the negative class. The curves show that cache hits can he detected with near perfect precision up to moderate recall scores. Note that our cache hit procedure attempts to produce cache hits but cannot guarantee cache hits (e.g., due to server routing), so some times in the cache hit distribution may actually be cache misses, which would hurt recall scores.
14

---

Auditing Prompt Caching in Language Model APIs
Table P-values from level of our audits: same prompt, per-user caching. Each column shows one combination of NUMVICTIMRE: QUESTS and timing source (client-side or server-side timing). Green indicates a significant p-value, after performing the appropriate Bonferroni corrections. Red indicates p-value that is not significant. indicates that the given timing source was not available for the API. APIs are grouped by whether caching was detected in this level and sorted alphabetically within groups.
NUMVICTIMREQUESTS
25
Provider
Model
Client
Server
Anthropic
Claude Haiku
7.8 10 10-21
Azure
rext-embedding-3--ae
1.7 10-42
Deep Infra
Llama 3.1 8B Instruct
9.5 x 10-116
Fireworks
Llama 3.1 8B Instruct
2.0 x 10-80
4.7 X 10 109
Lepton
Llama 3.1 8B Instruct
2.2 x 10-138
2.2 X 10-138
OpenAl
GPT-4o mini
2.4 x 10-66 66
2.9 X 10 10-105
OpenAl
sexteebbedding---mmaad
7.6 10-9
2.3 10~10
Perplexity
Llama 3.1 8B Instruct
1.9 10-90
Replicate
Llama 8B Instruct
2.2 x 10-140
Amazon
Claude Haiku
0.27
0.51
Azure
GPT-4o mini
0.95
Cohere
Command R
0.62
0.72
Cohere
embed-english-v3..
0.41
0.56
DeepSeek
DeepSeek Chat
0.75
Google
Gemini 1.5 Flash
0.17
0.20
Google
text-embedding-
0.20
0.24
Groq
Llama 8B Instruct
0.41
0.51
Hyperbolic
Llama 3.1 Instruct
0.72
Mistral
Mistral Nemo
0.56
0.96
Mistral
Mistral Embed
0.67
0.91
OctoAI
Llama 3.1 8B Instruct
0.32
0.27
Together
Llama 3.1 8B Instruct
0.51
0.96
Table P-values from level 2 of our audits: prompts with the same prefix but different suffixes, per-user caching. Each column shows one combination of VITTIREIRRQQ
timing source (client-side or server-side timing). Green indicates a significant p-value, indicates that the given
after performing the appropriate Bonferroni corrections. Red indicates p-value that is not significant. timing source was available for the API. A blank cell indicates that the given value of VUMVICTIMREQQEETT was tested because caching was detected in the API using smaller value of NuM NuMVIccIMREQQeESS. Caching was detected in all APIs audited in this level. APIs are sorted alphabetically.
NUMVICTIMREQQESTT
25
Provider
Model
Client
Server
Client
Server
Client
Server
Anthropic
Claude Haiku
9.6 10-37
Azure
moxt-embedding---sse 3-maall
0.20
6.0 10-1
6.9 x 10-42
Deep Infra
Llama 3.1 8B Instruct
0.03
5.0 10-22
Fireworks
Llama 3.1 8B Instruct
4.3 x 10-15
5.0 x 10 10-33 33
Lepton
Llama 8B Instruct
1.00
0.96
7.7 10 10
7.7 10-10
OpenAl
GPT-4o mini
9.5 x 10
1.5 x 10 10-39
OpenAl
(ext-embedding-3-smu
0.03
0.03
0.10
0.17
2.6 10-12
4.3 10-16
Perplexity
Llama 3.1 8B Instruct
5.4 x 10 68
Replicate
Llama 8B Instruct
8.6 10 160
15

---

Auditing Prompt Caching in Language Model APIs
Table P-values from level of audits: prompts with the prefix but different suffixes, per-organization eaching. Each column shows combination of NUM VICTIIIRREEU and timing source (client-side server-side timing). Green indicates a significant p-value, after performing the appropriate Bonferroni corrections. Red indicates p-value that is not significant. "_" indicates that the given timing source was not available for the API. A blank cell indicates that the given value of NUM VICTIMMREQUSSSS was not tested because caching was detected API using a smaller value NuMVIcCIMREQQUEST. Caching was detected in APIs audited this level. APIs are sorted alphabetically.
NUMVICTIMREQUESTS
5
25
Provider
Model
Client
Server
Client
Server
Client
Server
Anthropic
Claude 3 Haiku
1.7 x 10
Fireworks
Llama 3.1 8B Instruct
x 10-21
5.2 32
OpenAI
GPT-4o mini
1.1 x 10-19
4.6 x 10-34
OpenAI
(exteembedding---snan
0.27
0.14
0.27
0.27
8.2
10-14
x 10-14
Table P-values from level 4 of our audits: prompts with the same prefix but different suffixes, glohal cache sharing. Each column shows one combination NUMVICTIMREQUESTS and timing source (client-side or server-side timing). Green indicates a significant p-value, after performing the appropriate Bonferroni corrections. Red indicates p-value that is not significant. "_" indicates that the given timing source was not available the API. A blank cell indicates that given value NUM VICTIMRQUUESSS was not tested because caching was detected in the API using smaller value of NuMVICTIMREQQUESTS APIs are grouped hy whether caching was detected in level sorted alphabetically within the groups
NUMVICTIMREQQESTTS
25
Provider
Model
Client
Server
Client
Server
Client
Server
text-embedding-3-sma.
0.16
x 21
Deep Infra
Llama 3.1 8B Instruct
6.5 x 10
7.5 x 10
Fireworks
Llama 8B Instruct
9.0 x 10-17
5.2 10 32
Lepton
Llama 3.1 8B Instruct
0.12
0.07
1.2 x 10
1.4 10-9
OpenAl
text-embedding---nne
0.41
0.20
X 10-19
x 10-19
Perplexity
Llama 3.1 8B Instruct
5.3 x 10-74
Replicate
Llama 8B Instruct
8.6 8.6 150
Anthropic
Claude Haiku
0.24
0.77
0.87
OpenAl
GPT-4o mini
0.41
0.20
0.41
0.62
0.41
0.94
16

---

Auditing Prompt Caching in Language Model APIs
Table Server-side response times and first five embedding coordinates when sending the same prompt 25 consecutive times to the OpenAl text-embedding smalll API from the user. Blue denotes the "normal" embedding returned in most of the responses with normal response times, which indicate cache misses. Green denotes fast response times, which indicate cache hits, Red denotes embeddings that differ from both the "normal" and "cache hit" embeddings.
Embedding
Time (s)
Coordinate 1
Coordinate
Coordinate 3
Coordinate
Coordinate
0.100
0.00522740
0.02509154
-0.04450446
0.01837845
0.02944954
0.096
0.00534875
0.02516800
0.04450600
0.01841488
0.02952400
0.119
0.00534875
0.02516800
00.04450600
0.01841400
0.02952400
0.088
0.00534875
0.02516800
0.04450600
0.01841400
0.02952400
0.216
0.00523302
0.02509207
0.04457144
0.01835683
0.02947217
0.100
0.00534875
0.02516800
0.04450600
0.01841400
0.02952400
0.096
0.00534875
0.02516800
-0.04450600
0.01841400
0.02952400
0.088
0.00534875
0.02516800
-0,04450600
0.01841400
0.02952400
0.077
0.00534875
0.02516800
-0.04450600
0.01841400
0.02952400
0.036
0.00535751
0.0251 7040
0.044555426
0.01839376
0.029548822
0.076
0.00534875
0.02516800
-0.04450600
0.01841 400
0.02952400
0.124
0.00534875
0.02516800
-0.04450600
0.01841400
0.02952400
0.280
0.00522179
0.02509099
-0.04525511
0.01835604
0.02947091
0.032
0.00535751
0.02517040
-0.04455426
0.01839376
0.029548822
0.089
0.00534875
0.02516800
0..44456000
0.01841400
0.02952400
0.034
0.00535751
0.02517040
-0.04455426
0.01839376
0.02954882
0.092
0.00534875
0.02516800
0..04450600
0.0184140 400
0.02952400
0.089
0.00534875
0.02516800
-0.04450600
0.018 400
0.02952400
0.103
0.00534875
0.02516800
0.04450600
0.01841 400
0.02952400
0.127
0.00523272
0.025134666
-0.04454690
0.01837780
0.029448499
0.094
0.00534875
0.02516800
-0.04450600
0.01841400
0.02952400
0.039
0.00535751
0.0251 7040
-0.04455426
0.01839378
0.029548822
0.035
0.00535751
0.02517040
-0.04455426
0.01839376
0.029548822
0.080
0.00534875
0.02516800
-0.04450600
0.01841400
0.02952400
0.034
0.00535751
0.02517040
-0.04455426
0.01839376
0.02954882
17

---

Auditing Prompt Caching in Language Model APIs
Table Server-side response times and first five embedding coordinates when sending the same prompt 25 consecutive times to the OpenAl text-embedding-1 small API from the user. Blue denotes the "normal" embedding returned in most of the responses with normal response times, which indicate cache misses. Green denotes fast response times, which indicate cache hits, Red denotes embeddings that differ from both the "normal" and "cache hit" embeddings.
Embedding
Time (s)
Coordinate 1
Coordinate
Coordinate 3
Coordinate
Coordinate
0.093
0.00455398
0.02148935
-0.05159185
0.01970582
0.02810146
0.079
0.00455398
0.021 48935
-0.05159185
0.01970582
0.028101
0.081
0.00455398
0.021489355
-0.05159185
0.01970582
0.02810145
0.087
0.00455398
0.021 48935
-0.05159185
0.01970582
0.0281014
0.112
0.00455518
0.02146046
0..0115746
0.01972101
0.02807036
0.038
0.00453244
0.02149035
0.05159424
0.01968498
0.02810276
0.113
0.00455398
0.02148935
-0.51559885
0.01970582
0.02810146
0.036
0.00453244
0.02149035
-0.05159424
0.01968498
0.02810276
0.078
0.00455398
0.02148935
-0.05159185
0.01970582
0.02810146
0.118
0.00455531
0.02148279
-0.05153262
0.01974330
0.02809289
0.079
0.00455518
4.02146046
-0.05157467
0.01972101
0.02807036
0.084
0.00455398
0.02148935
-0.05159185
0.01970582
0.02810148
0.096
0.00455398
0.02148935
0..5159185
0.01970582
0.0281014 46
0.110
0.00455398
0.02148935
-0.05159185
0.01970582
0.028101 10146
0.089
0.00455398
0.02148933
0.05159185
0.01970582
0.02810) 46
0.063
0.00453244
0.02149035
0.051559424
0.01968498
0.02810276
0.035
0.00453244
0.02149005
0.05159424
0.01968498
0.02810276
0.094
0.00455398
0.021 48935
-0.05159885
0.01970582
0.02810146
0. 0.100
0.00455398
0.021 48935
0.05159185
0.01970582
0.02810146
0.033
0.00453244
0.02149035
0.051559424
0.01968499
0.02810276
0.112
0.00455398
0.02148935
-0.05159185
0.01970582
0.02810146
0.036
0.00453244
0.02149035
-0.05159424
0.01 401968498
0.02810276
0.033
0.00453244
0.02149035
-0.05159424
0.01968498
0.02810276
0.092
0.00455398
0.02148935
-0.05159185
0.01970582
0.02810146
0.118
0.00454002
0.02149847
-0.05158763
0.01983471
0.02807742
18

---

Auditing Prompt Caching in Language Model APIs
Table Server-side response times and first five embedding coordinates when sending the same prompt 25 consecutive times to the OpenAl text-embedding-1 small API from the user. Blue denotes the "normal" embedding returned in most of the responses with normal response times, which indicate cache misses. Green denotes fast response times, which indicate cache hits, Red denotes embeddings that differ from both the "normal" and "cache hit" embeddings.
Embedding
Time (s)
Coordinate 1
Coordinate
Coordinate 3
Coordinate
Coordinate
0.113
0.00306934
0.02534029
-0.056561 14
0.02567696
0.02828057
0.033
0.00308367
0.02534279
0.0565218 82
0.025701
0.02821 602
0.087
0.00306934
0.025340299
0.0565511
0.02567696
0.02828057
0.097
0.00306934
0.02534029
0.0565611
0.02567696
0.02828057
0.163
0.00306934
0.02534029
0.05656114
0.02567696
0.02828057
0.142
0.00305834
0.02536441
0.05647554
0.02572376
0.02826022
0.033
0.00308367
0.02534279
0.05655182
0.02570194
0.02821602
0.119
0.00306308
0.02531247
0.05650425
0.02569395
0.02827457
0.090
0.00306934
0.02534029
-0.05656 114
0.02567696
0.02828057
0.100
0.00306934
0.02534029
0.05656114
0.02567696
0.02828057
0.261
0.00306656
0.02531800 808
-0.05647188
0.02576698
0.02823594
0.426
0.00308757
0.02537424
0.05566312
0.02575598
0.02822604
0.040
0.00308367
0.02534279
0.05652182
0.02570194
0.02821 602
0.036
0.00308367
0.02534279
0.05652182
0.02570194
0.02821602
0.075
0.00306934
0.02534029
0.05656111 14
0.02567696
0.02828057
0.087
0.00306934
0.025340299
0.05656114
0.02567696
0.02828057
0.093
0.00306934
0.025340299
0.05656114
0.02567696
0.02828057
0.116
0.00305834
0.02536461
0.05647554
0.02572376
0.02826022
0.140
0. 1.00306308
0.02531247
0.05650425
0.02569395
0.02827457
0.034
0.00308367
0.02534279
00.5655288
0.02570194
0.02821602
0. 0.033
0.00306686
0.025365400
-0.05638750
0.02572453
0.02823864
0.033
0.00308367
0.02534279
-0.05652182
0.02570194
0.02821 602
0.032
0.00308367
0.02534279
-0.05652182
0.02570194
0.02821602
0.089
0.00306934
0.02534029
-0.05656114
0.02567696
0.02828057
0.032
0.00308367
0.02534279
-0.05652182
0.02570194
0.02821602
19

---

Auditing Prompt Caching in Language Model APIs
(a) length (same prompt))
(b) Vary length (same prefix, difi. suffix)
Very prefix match length
Vary model
0.6
0.6
evvalue
0.
evalue
0.4
0.4
0.4
XA
0.4
0.2
0.2
02-
10:00
2000
3000
4003
5000
1000
2000
3000
4000
5000
250
400
620
600
1000
18
68
118
700
50B
4058
Promot length (tokens)
Prompt length (tokens)
Prefix match length (tokens)
Liama model size
(a) Vary prompt length (same prompn))
(b) Vary prompt length (same prefix, diff. suffid)
Vary prefix mitch length
Vary model size
108
ovalue (log scale)
10.93
prvallve (log sicate
Divalue sccale)
X
10-41
20-11
10-5
2p-15
20-19
10-1
10-10
10-10
12-2te
20-120
10:46
13:45
1000
2000
2000
4003
5000
1000
2000
3000
4000
5000
200
400
600
800
1000
15
68
118
705
508 4038
Prompt lengthi (takens)
Prompt longth (tokens)
Proffx match length (tokens)
Liama model size
Fireworks iclient time)
Firewarks (server time)
Perplexity (client time)
Replicate (server time)
Figure 7. Ablations on the effects of PROMPTLEENTTE PREfXFRACTION, and model size on the audit p-values. Each test is run using NUMSAMPLES 250. The top and hottom display the p-values linear and logarithmic scales, respectively. In (a)-(c), the prompt length or prefix match length decreases, the p-values grow larger. In (d), we detect caching across all model sizes, with no clear relationship between model size p-values.
20


Reviews:
[{"review_id": "X09DSrGXg3", "reviewer": "Reviewer_6FQk", "summary": "This paper presents an empirical audit of prompt caching mechanisms in language model APIs. It demonstrates that timing differences, arising from cache hits and cache misses, can potentially leak private information and even reveal details about a model’s architecture. The study employs statistical hypothesis testing across various real-world API providers to characterize different levels of cache sharing (per-user, per-organization, and global). The results indicate significant vulnerabilities, including potential privacy leakage and exposure of proprietary model details.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed and thoughtful review! We address the main points below.\n\n## Random prompts produce cache misses\n\nWe assume that random prompts produce cache misses because it is exceedingly unlikely that a random prompt shares a prefix of noticeable length with any cached prompts. In the worst case, assume that all other users are sending random prompts with the same structure as in the paper, i.e., random letters separated by spaces. (In reality, very few, if any, other users will be sending such prompts.) As mentioned in the paper (page 4, line 196, left column), the probability that two of these random prompts share a prefix of 15 tokens or longer is less than $10^{-25}$. Assume that the server can store 1 billion prompts in its cache (in reality, the true cache capacity is likely much smaller). Then, using a union bound, the probability that a random prompt shares a prefix of 15 tokens or longer with any cached prompt is less than $10^{-25} \\times 10^{9} = 10^{-16}$. We send 250 random prompts for cache miss timings in each audit, so using another union bound, the probability that any of these prompts produce cache hits is less than $10^{-12}$.\n\nWe are also able to empirically confirm that random prompts produce cache misses. Some API providers have officially released prompt caching features, such as [OpenAI](https://platform.openai.com/docs/guides/prompt-caching) and [Anthropic](https://docs.anthropic.com/en/docs/build-with-claude/prompt-caching). As part of these features, each API response returns the number of cached prompt tokens. Using this information, we ran some simple tests that empirically confirm that the random prompts used in our experiments consistently produce cache misses in real-world API usage scenarios.\n\n## Random prompts versus natural language\n\nWe used random alphabetic prompts instead of natural language prompts for a few reasons. We can assume that random prompts reliably produce cache misses, as discussed above. However, there is a greater chance that prefixes of realistic natural language prompts have already been sent by other users, making it harder to reliably measure cache misses. In addition, it is more difficult to construct a clean, well-defined distribution of natural language prompts containing exactly `PromptLength` tokens, compared to using random prompts. This distribution is important for our rigorous statistical hypothesis testing framework.\n\nAs mentioned in the paper (page 6, line 302, right column), our audits detected the exact level of cache sharing stated by OpenAI and Anthropic for their chat models (per-organization sharing, but not global sharing). This demonstrates the efficacy of our audits on real-world APIs, even if the random prompts do not necessarily reflect realistic workflows.\n\n## Real-world exploitation and mitigations\n\nAs discussed in Section 4.4: Difficulty of Prompt Extraction Attacks (line 350, page 7, left column), we believe that real-world exploitation of these vulnerabilities is challenging. However, we believe that even potential privacy leakage due to global cache sharing is a cause for concern, especially as LLM APIs are being used by increasingly many users and companies for increasingly many tasks, which may include sensitive data. In addition, future work may overcome or eliminate the practical challenges we discussed.\n\nTo mitigate these vulnerabilities, we believe that API providers should disable global cache sharing and disclose the level of cache sharing. Following our responsible disclosure, several API providers followed this approach. OpenAI, Microsoft Azure, and Fireworks AI worked quickly to mitigate vulnerabilities by stopping global cache sharing. (Note that this is not an exhaustive list of all companies that implemented fixes.) Fireworks AI also added detailed [documentation](https://docs.fireworks.ai/guides/prompt-caching) about prompt caching and data privacy, as well as an option to opt-out of caching for a particular prompt. The mitigations implemented by these companies illustrate the real-world impact of our findings.\n\nAlso, we found that prompt caching can leak information about model architecture, which is of practical importance given the competitiveness and secrecy of the modern LLM landscape. Namely, we found evidence that OpenAI’s text-embedding-3-small model has a decoder-only Transformer architecture, which was previously not publicly known.\n\n## Ethics review\n\nAs discussed in the Impact Statement (page 9, line 440, left column), to mitigate real-world harms arising from our research, we followed standard responsible disclosure practices for security vulnerabilities. In October 2024, we disclosed our audit results with each API provider in which we detected prompt caching. We gave providers 60 days to address the vulnerabilities before publicly releasing or submitting our findings, and the actual time elapsed ended up being longer.", "claims_and_evidence": "The paper makes strong claims regarding the vulnerability of LLM APIs to timing side-channel attacks via prompt caching. It supports these claims with rigorous statistical evidence. However, some evidence is derived under assumptions, such as using random prompts to simulate cache misses, that may not universally hold in real-world deployments. This reliance on assumptions could limit the generalizability of the findings, and further empirical validation is needed.", "methods_and_evaluation_criteria": "The methodology, including the construction of cache hit and miss procedures and the application of nonparametric statistical tests, is innovative and well-structured for the research problem. Nevertheless, the evaluation relies on assumptions (e.g., that random prompts always produce cache misses) that are difficult to verify. Additional experiments using more realistic or varied prompt distributions, along with evaluation criteria that mimic real-world conditions, would strengthen the paper’s claims.", "theoretical_claims": "While the paper does not focus heavily on formal proofs, it builds a insightful theoretical basis for linking cache timing differences to potential privacy leaks.", "experimental_designs_or_analyses": "The experimental design is methodically sound, with clear separation between the cache hit and miss procedures and extensive use of statistical tests. However, the reliance on a synthetic prompt distribution (i.e., random sequences of tokens) to simulate cache misses is a weakness. It remains unclear whether these conditions reflect typical user inputs. The experiments would be more convincing if supplemented with tests involving natural language prompts or real-world API logs, which could validate the assumptions underlying the timing measurements.", "relation_to_broader_scientific_literature": "The paper positions itself well within the landscape of cache timing attacks and side-channel vulnerabilities. It is also motivated by recent developments in LLM optimization and inference acceleration.", "essential_references_not_discussed": "The mentioned related works are essential to understanding the main contributions of this paper.", "other_strengths_and_weaknesses": "Strengths:\n\n- The paper tackles an important and timely security issue in AI.\n- The statistical framework is rigorous, and the empirical analysis is detailed.\n- The categorization of cache sharing levels is insightful and adds nuance to the discussion.\n\nWeaknesses:\n\n- Heavy reliance on assumptions regarding prompt distributions (e.g., that random prompts always yield cache misses.\n- The experimental setup lacks naturalistic workloads that reflect how APIs are used in practice.\n- Some discussion could be more in-depth, particularly with regard to potential exploitation scenarios and mitigation strategies.", "other_comments_or_suggestions": "I recommend including an explicit ethical statement that addresses the potential misuse of the research findings. Additionally, integrating a prototype or a detailed case study that illustrates a practical exploitation scenario would greatly enhance the paper’s impact. Finally, further empirical validation of the assumptions regarding prompt distribution and cache behavior is necessary to reinforce the generality of the conclusions.", "questions_for_authors": "- Can you provide empirical evidence that the assumption of random prompts reliably produces cache misses holds true in real-world API usage scenarios?\n\n- Have you considered evaluating your attack model using natural language prompts or actual API traffic to better mimic realistic conditions?\n\n- Would the authors be able to include a proof-of-concept demonstration or a detailed case study that illustrates a potential real-world exploitation of these vulnerabilities?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_flag": "Flag this paper for an ethics review.", "ethics_expertise_needed": ["Responsible Research Practice (e.g., IRB, documentation, research ethics, participant consent)"], "ethical_review_concerns": "The paper highlights significant privacy vulnerabilities and the potential for misuse of timing side-channel attacks, which may allow attackers to infer sensitive information from user prompts."}}, {"review_id": "LMhLTI43yb", "reviewer": "Reviewer_uAtv", "summary": "The paper explores a novel privacy risk for hosted language models: timing attacks on the host's prefix cache. They use hypothesis testing to demonstrate that popular LM providers are indeed using prefix caching.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed and thoughtful review! We address the questions below.\n\n## Speculative decoding\n\nThis is an interesting point. We have thought about this, and we believe that speculative decoding should not impact the TTFT when we set the max response tokens to 1. First, we note that speculative decoding is not beneficial when only 1 response token is generated. Speculative decoding is only beneficial when the smaller draft model can generate multiple tokens, and the larger target model can “check” these tokens in parallel. Since the larger target model has to make a forward pass regardless, when generating only 1 token, it is faster to skip the smaller draft model.\n\nIf speculative decoding is nevertheless enabled when generating only 1 token, we believe that it would not cause a noticeable variation in TTFT across different prompts (of the same length). In this scenario, the smaller draft model would generate 1 token, then the larger target model would make a forward pass and either accept or reject the draft token. The only timing difference between accepting and rejecting would come from resampling a token from the target distribution, which is negligible compared to the time for the forward pass.\n\nNote that when the LLM generates full responses (number of tokens $\\gg 1$), speculative decoding causes data-dependent timing variations that may be exploited. As mentioned in the related works (page 8, line 404, right column), Carlini & Nasr (2024) and Wei et al. (2024) exploit speculative decoding to extract encrypted and streamed LLM responses by measuring delays between packets.\n\n## Intentionally delaying responses\n\nYes, we believe that intentionally delaying the response times for cache hits so that they look like cache misses is a viable mitigation for providers. We briefly touch upon this in the paper (page 8, line 410, left column). This eliminates the benefits of prompt caching for users, but API providers could still benefit, as cached prompts require less GPU processing time.\n\nProviders would need to be somewhat careful about implementing this, as simply waiting a random amount of time may not adequately disguise cache hits. One better strategy is to first compute distributions of the server-side TTFT for cache misses for various prompt lengths. Then, when a cache hit occurs, the server would sample a TTFT from the cache miss distribution corresponding to the given prompt length, and delay the response until that TTFT has elapsed (if the actual TTFT has already exceeded the sampled TTFT, then send the response immediately). This way, the distribution of times for cache hits and cache misses should approximately match each other.", "claims_and_evidence": "I believe all of the claims made are well supported with statistical evidence and findings that are not completely certain are appropriately presented.", "methods_and_evaluation_criteria": "The method proposed looks like exactly the right thing to do and the evaluation of the test on popular LM providers is of great significance to the community.", "theoretical_claims": "There are no major theoretical claims made.", "experimental_designs_or_analyses": "The experimental design for public LM api providers is very sound. One potentially interesting result that is not included is testing in a self-hosted API server which can be configured to support prefix caching (like the one provided by VLLM). This may help demonstrate how easy it is to detect prefix caching in an ideal setting.", "relation_to_broader_scientific_literature": "This paper is the latest in a line of work on attacking models via their APIs, many of which are detailed in the related work section. The specific attack of this paper has not been addressed by any prior work. It represents a significant new attack which will hopefully influence model providers to carefully design their caching systems with properly enforced boundaries.", "essential_references_not_discussed": "I am not aware of any essential works that were not cited properly in the paper.", "other_strengths_and_weaknesses": "Other strength: The framing of audit in terms of hypothesis testing is very natural!", "other_comments_or_suggestions": "The authors responsible disclosure of the detected vulnerability to the model providers is commendable.", "questions_for_authors": "Another technique that may affect the TTFT is speculative decoding. Do you think there is any potential to try to infer the speculation behavior in order to help \"denoise\" the timing results and focus on the impact of the cache?\n\nDo you think LM providers could strategically delay their request responses so the timing is identical between cached and uncached requests, while still saving on compute cost?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FfQZkacdpY", "reviewer": "Reviewer_KxRJ", "summary": "This paper investigates the privacy leakage caused by the prompt caching in LLMs. Basically, prompt caching improves the efficiency of inference by caching and reusing the internal results of previous prompts. The attack could infer whether a given prompt has been used (cached) by simply checking the time to the first token (TTFT).\nThe paper provides an extentive evaluation to support its claim. It also proposes some mitigation strategies which are reasonable and insightful.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed and thoughtful review! We address the main points below.\n\n## Practical exploitability\n\nWe agree that practical exploitations of prompt cache sharing are challenging. The attacker needs to guess a long prompt prefix to check if it is cached, as you described. As we discussed in Section 4.4: Difficulty of Prompt Extraction Attacks (page 7, line 350, left column), a natural idea is to use breadth-first search to try to extract cached prompts token-by-token. However, we were unable to execute practical prompt extraction attacks due to several challenges, such as the difficulty of making repeated measurements. Accordingly, we emphasized that the privacy leakage is only potential throughout the paper.\n\nHowever, we believe that even potential privacy leakage due to global cache sharing is a cause for concern, especially as LLM APIs are being used by increasingly many users and companies for increasingly many tasks, which may include sensitive data. In addition, future work may overcome or eliminate the practical challenges we discussed. \n\nFollowing our responsible disclosure, several API providers made changes to mitigate the potential privacy leakage. OpenAI, Microsoft Azure, and Fireworks AI worked quickly to mitigate vulnerabilities by stopping global cache sharing. (Note that this is not an exhaustive list of all companies that implemented fixes.) Fireworks AI also added detailed [documentation](https://docs.fireworks.ai/guides/prompt-caching) about prompt caching and data privacy, as well as an option to opt-out of caching for a particular prompt. The mitigations implemented by these companies illustrate the real-world impact of our findings.\n\nAlso, we found that prompt caching can leak information about model architecture, which is of practical importance given the competitiveness and secrecy of the modern LLM landscape. Namely, we found evidence that OpenAI’s text-embedding-3-small model has a decoder-only Transformer architecture, which was previously not publicly known.", "claims_and_evidence": "Most claims made in this paper are reasonable and supported by existing works or evaluation results.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "There is no proof in this paper. It is mostly an empirical paper based on existing statistical methods.", "experimental_designs_or_analyses": "The experimental designs and analyses make sense to me.", "relation_to_broader_scientific_literature": "This paper provides a thorough analysis of privacy and security vulnerabilities associated with prompt caching in LLM APIs. It effectively highlights potential risks and discusses mitigation strategies. The findings have significant implications, raising awareness about security concerns in model deployment and emphasizing the need for preventive measures.", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "Strengths:\n\n- The code and experimental data are provided.\n- The writing is good and easy to follow.\n\nWeakness:\n\n- Limited discussion of practical exploitability of the proposed leakage.", "other_comments_or_suggestions": "This paper explores a novel privacy leakage lead by the prompt caching techniques in most recent LLMs.\nThe claims are mostly correct and reasonable, supported by a series of evaluations.\n\nI appreciate the disclose of the potential issues and the discussion of countermeasures.\nMy primary concern is that the practical exploitability of the proposed leakage is not easy, as it's hard to guess out the prompt prefix with a nontrival percentage of the entire length. Therefore, the leakage may not be that significant if it is not easily exploited. I suggest the authors could pay some attention on this part.", "questions_for_authors": "Please see above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VOTuVABAMH", "reviewer": "Reviewer_iqr7", "summary": "This paper ascertains that hint caching in the LLM API can precipitate privacy breaches and divulge model architecture information. To prevent such issues, API providers are recommended to permit only user-level caching and make caching policies public for enhanced transparency. Through these examinations, the authors aim to continue evaluating and auditing the security and privacy of machine learning systems. Using empirical research, the paper reveals the risks of prompt caching and proposes corresponding mitigation strategies, providing a valuable reference for related work.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed and thoughtful review! We address the main points below.\n\n## 1. Request routing strategies\n\nWe can show that our experimental results are valid regardless of routing strategies. Under the null hypothesis $H_0$ of no caching, request-routing will be independent of whether the prompt was recently sent. Therefore, since the distribution of prompts $\\mathcal{P}$ is the same in the cache hit and cache miss procedures (these procedures differ only in whether the attacker’s prompt was previously sent), a non-caching request-routing strategy will perform the same on the distributions of the cache hit and cache miss procedures. Therefore, $\\mathcal{D}\\_\\text{hit} = \\mathcal{D}\\_\\text{miss}$ still holds under the null hypothesis $H_0$, so the statistical audit is valid.\n\nIf the API provider is performing caching, then different routing strategies may make it easier or harder to detect caching, but the audit still outputs valid p-values with respect to the null hypothesis. For example, if prompts are intentionally routed to a server where the prompt is already cached, it will be easy to produce and detect cache hits using `NumVictimRequests = 1`. On the other hand, if prompts are randomly routed, more victim requests may be needed to detect caching. Multiple victim requests may be needed to cache the prompt in enough servers for the attacker’s prompt to have a sufficient probability of producing a cache hit.\n\n## 2. Bonferroni correction\n\nWhile the Bonferroni correction is indeed conservative, it is simple and straightforward, and we only use a maximum correction factor of 6. We are confident that performing Bonferroni correction did not cause our audits to miss any real caching because the p-values for the APIs in which we did not detect caching were orders of magnitude larger than the significance level of $\\alpha = 10^{-8}$. As shown in Table 3 in the appendix (page 14), in the first level of audits, all non-significant p-values were larger than 0.1, and most significant p-values were many orders of magnitude smaller than $10^{-8}$.\n\n## 3. Other factors that could cause similar caching phenomena\n\nFor the purposes of our audit, the specific caching mechanism is unimportant, as long as it follows the simple properties we describe (page 2, line 70, right column). The potential privacy leakage does not depend on what specific mechanisms cause timing differences between cached and non-cached prompts; the leakage occurs simply because there *are* timing differences. Even if the caching phenomena has strange causes—e.g., the server intentionally delays responses for new prompts but immediately returns responses for previously seen prompts—our audits can detect it, and the timing differences lead to potential privacy leakage.", "claims_and_evidence": "The comparative experiments are relatively comprehensive.", "methods_and_evaluation_criteria": "The method proposed in this paper can be used to detect the existence of prompt cache and the level of cache sharing. This approach is useful for understanding the behavior of LLMs in real-world applications, especially in privacy-focused scenarios, where the audits mentioned in the paper are conducted over multiple actual API calls, suggesting that the approach is closely related to real-world application scenarios.", "theoretical_claims": "The article basically does not involve theoretical proof.", "experimental_designs_or_analyses": "The paper conducts experiments on prompt caching in language model APIs. It targets 17 API providers, constructs cache - hit and cache - miss procedures with different parameters, uses statistical hypothesis testing and two - sample KS test to determine caching status, cache - sharing levels, evaluate attackers' discrimination ability, and explore information leakage.", "relation_to_broader_scientific_literature": "The key contribution of this paper is the development of an audit method to detect the phenomenon of prompt caching in language model apis and identify different levels of prompt sharing patterns, including global, organizational, and individual levels, providing a new perspective on the design and implementation of language model APIs.", "essential_references_not_discussed": "Not yet.", "other_strengths_and_weaknesses": "**Strengths**：\n1. The paper centers on the significant but overlooked prompt caching in language model APIs, exploring privacy and architecture info leakage.\n2. Its well - designed cache - hit/miss experiments with statistical testing can precisely identify API caching and sharing levels, offering a practical security - detection tool.\n3. The experiments involve diverse models, representing different APIs' prompt - caching realities.\n\n**Weaknesses**：\n1. Complex request - routing strategies may distort experimental results and mislead caching judgments.\n2. The Bonferroni correction in multiple testing is too conservative, potentially missing real caching and underestimating security risks.\n3. The paper fails to deeply verify other factors that could cause similar caching phenomena.", "other_comments_or_suggestions": "Not yet.", "questions_for_authors": "Not yet.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "Not yet."}}]

Author rebuttal:

Meta-review: This paper introduces a timely and impactful empirical audit of prompt caching behavior in commercial LLM APIs, revealing potential privacy and security vulnerabilities through carefully designed timing-based side-channel analysis. The methodology is rigorous and well-motivated, grounded in statistical hypothesis testing, and the evaluation is thorough across a broad range of providers. Despite relying on assumptions such as random prompts generating cache misses, the authors provide both theoretical justifications and empirical confirmation that lend credibility to their setup. The work has already prompted real-world impact through responsible disclosure, with several providers implementing mitigations in response. While the practical exploitability of the described leakage may be limited today, the paper makes a compelling case that even potential leakage is significant as LLMs become more widely used in sensitive settings. Overall, despite some limitations in assumptions and generalizability, the contributions are novel, the findings are important, and the paper should be included in the conference.
Decision: Accept (poster)