Logits of API-Protected LLMs Leak Proprietary Information
Authors: Matthew Finlayson
Xiang Ren
Swabha Swayamdipta
Venue: COLM (2024)
Keywords: softmax bottleneck, model stealing, API, language models, large language models
TL;DR: We show how to extract non-public information about API-protected LLMs from a few queries, including the embedding size and output space of the model, then investigate useful applications for adversarial attacks and model accountability.

Abstract: Large language model (LLM) providers often hide the architectural details and parameters of their proprietary models by restricting public access to a limited API. In this work we show that, with only a conservative assumption about the model architecture, it is possible to learn a surprisingly large amount of non-public information about an API-protected LLM from a relatively small number of API queries (e.g., costing under $1000 USD for OpenAI’s gpt-3.5-turbo). Our findings are centered on one key observation: most modern LLMs suffer from a softmax bottleneck, which restricts the model outputs to a linear subspace of the full output space. We exploit this fact to unlock several capabilities, including (but not limited to) obtaining cheap full-vocabulary outputs, auditing for specific types of model updates, identifying the source LLM given a single full LLM output, and even efficiently discovering the LLM’s hidden size. Our empirical investigations show the effectiveness of our methods, which allow us to estimate the embedding size of OpenAI’s gpt-3.5-turbo to be about 4096. Lastly, we discuss ways that LLM providers can guard against these attacks, as well as how these capabilities can be viewed as a feature (rather than a bug) by allowing for greater transparency and accountability.

Published as a conference paper at COLM 2024
Logits of API-Protected LLMs Leak Proprietary Information
Matthew Finlayson Xiang Ren Swabha Swayamdipta
Thomas Lord Department of Computer Science University of Southern California {mfinlays, xiangren, swabhas]@usc.ed.
Abstract
Large language model (LLM) providers often hide the architectural details and parameters of their proprietary models by restricting public access to limited API. In this work we show that, with only a conservative as- sumption about the model architecture, it is possible to learn a surprisingly large amount of non-public information about an APl-protected LLM from a relatively small number of API queries (e.g., costing under $1000 USD for OpenAl's gpt-3. 5- 5-turbo). Our findings are centered on one key ob- servation: most modern LLMs suffer from softmax bottleneck, which restricts the model outputs to linear subspace of the full output space. We exploit this fact to unlock several capabilities, including (but not limited to) obtaining cheap full-vocabulary outputs, auditing for specific types of model updates, identifying the source LLM given a single full LLM output, and even efficiently discovering the LLM's hidden size. Our empirical investigations show the effectiveness of our methods, which allow us to estimate the embedding size of Open gpt- 5-turbo to be about 4096. Lastly, we discuss ways that LLM providers can guard against these attacks, well as how these capabilities can be viewed as a feature (rather than a bug) by allowing for greater transparency and accountability.
1 Introduction
As large language models (LLMs) become more capable and valuable, many LLM providers have shifted toward training closed-source production LLMs,, accessible only via (paywall- restricted) APIs to protect trade secrets (e-g., AT et al., 2024). As useful feature, these APls often allow users to access the probabilities (or logits, ie., unnormalized scores) that the LLM assigns to specific tokens. It turns out that such an interface reveals much more information about the underlying model than previously thought, including exposing non- public architectural details of the LLM. Closed APIs, therefore, may provide false sense of security to production LLM providers who may assume that their product information is private. At the same time, these capabilities empower the community with tools to audit LLM providers for bad behaviors such as unannounced model updates and abuse of open-source LLMs (Mökander et al., 2023).
In this paper we show that is possible to extract detailed information about LLM parame- terization using only common API configurations. Our findings are centered on one key observation: most modern LLMs suffer from softmax bottleneck (Yang et al., 2018), which restricts the model outputs to a linear subspace of the full output space, as illustrated in Figure 1. We call this restricted output space the LLM's image ($2). This image can be obtained by collecting a small number of L.LM outputs, and serves as unique identifier or signature for the model. We propose several algorithms that enable us to obtain the LLM image at low cost and speed for standard LLM APIs (53).
Furthermore, we show that LLM images are useful in several applications, revealing im- portant information about the model architecture. Concretely, we can exploit this image to provide an efficient algorithm for extracting full LLM outputs (54), to find the hidden dimension (embedding) size of the LLM ($5), to detect and identify different kind of updates made to the model and to attribute model outputs to specific model (66), among other
1

---

Published as a conference paper at COLM 2024
W
softmax
R
R
Ap
Figure 1: LLM outputs are constrained to low-dimensional subspace of the full output space. We can use this fact to glean information about APl-protecced LLMs by analyzing their outputs. Here we show how toy LLM'' low-dimensional embeddings I (illus- trated here 1-D space) are transformed linearly into logits in R" (here, 3D space) via the softmax matrix W. The resulting outputs lie within = 1)-dimensional subspace of the output space. We call this low-dimensional subspace the image of the model. We can obtain a basis for the image of an API-protected LLM by collecting of its outputs. The LLM's image reveal non-public information, such as the LLM's embedding size, but it can also be used for accountability, such as verifying which LLM an API is serving.
potential applications (S7). We demonstrate several of our proposed applications in order to empirically verify their effectiveness. Notably, we design and implement an algorithm for of
finding the embedding size of an API-based LLM, and use to estimate the embedding size
(a closed-source APl-protected LLM) to be 4096.
Overall, our proposed methods have benefits for both LLM providers and for their clients. LLM providers may their model images to establish unique identities for their models, thereby protecting their product and building trust with clients. Our proposed methods will also allow LLM clients to hold providers accountable for malicious behavior (Anderljung at. 2023). As concrete use case in accountabilitt, we demonstrate, using checkpoints from open-source LLMs that our LLM images can be used attribute LLM output probabilities (or logits) to their generating model with high accuracy. The sensitivity of our LLM image to slight changes in the LLM parameters also makes them suitable for inferring granular information about the specific type of model update.
Considering several proposals guard access LLM images, we find no obvious fix without dramatically altering the LLM architecture or making the API considerably less useful ($8). Providers who choose to alter their API to prevent LLM image access risk removing interfaces with valuable and safe use cases for LLM clients. Though our findings could be viewed as bug that LLM providers might feel compelled to patch, we prefer to view them as features that LLM providers may choose to keep in order to better maintain trust with their customers by allowing outside observers to audit their model. Ultimately, our results serve recommendation to LLM providers to carefully consider the consequences of their LLM architectures and APIs
This paper's contributions include
A method for extracting information about APl-protected models, including the model's output space and embedding size.
Methods for extracting full-vocabulary logprob outputs from LLM APIs.
An estimate of the embedding size of an APT-protected LLM
An accelerated logprob extraction algorithm based on the LLM image.
An exploration of several other applications of our method for model accountability.
Concurrently with our work, Carlini et al. (2024) propose very similar approach for exposing details of production LLMs, though with focus on defenses and mitigations against such attacks. The "final layer" that they extract in their attack corresponds to what we refer to in our paper as the model image. We view our papers as complementary, since our work emphasises practical applications of LLM images for better L.L accountability.
2

---

Published as a conference paper at COLM 2024
Softmax matrix
Probabilities
W C Rexd
0.6
boyfriend
Input
0.2
queen
Karma is my
Transformer
MatMul
SoftMax
0.1
thought
0.1
breeze
Embedding
Logits
0.0
acrobat
Ag
e R
L R
Figure 2: A typical language model architecture. After the input its processed by neural network, usually a transformer (Vaswani et al., 2017), into a low-dimensional embedding h, is multiplied by the softmax matrix W, projecting it linearly from R onto R" to obtain the logit vector L. The softmax function is then applied to the logit vector to obtain valid probability distribution over next-token candidates.
2 LLM outputs are restricted to low-dimensional linear space
The outputs of typical LLMs are naturally constrained to lie on d-dimensional subspace of the full output space (Yang et al., 2018; Finlayson et al., 2024). To understand this, begin by considering the architecture of a typical LLM (Figure 2). this architecture, a transformer' with embedding size d outputs low-dimensional contextualized embedding h E Rd (or simply embedding). Projecting the embedding onto RR via the linear map defined by the LLM's softmax matrix W, we obtain logits l = Wh.
Theorem 1 (Low-rank logits). LLM logits lie on a d-dimenssional subspace of R
Proof. Because W is in Rexd, its rank (i.e., number of linearly independent columns) is at most d. The rank of a matrix corresponds to the dimension of the image of the linear map defines, i.., the vector space comprising the set of possible outputs of the function. In other words, if the linear map w is defined as w(h) = Wh, then w's image im(w) = {whh)& e IR h Rdd is a d-dimensional subspace of IR".
Thus, the LLM's logits will always lie on the d-dimensional subspace of R
The low -ow-dimesiional restriction of logits actually translates into a similar restriction on LLM probabilities, which useful since most L.M APIs return hog-)probabilitien instead of logits. To understand why this is, we turn our attention to the model's next-token distribution p = softmax(E).
(). The softmax function transforms logits into a valid probability distribution, i.e,, a o-tuple of real numbers between 0 and whose sum is 1. The set of valid probability distributions over v items is commonly referred to the v-simplex, or Av.
Theorem 2 (Low-rank probabilities). LLM probabilities lie on d-dimensional subspace of Av-
Proof. Perhaps surprisingly, Ap is also valid (v 1i-dimensional vector space (albeit under non-standard definitions of addition and scalar multiplication) and the softmax function linear map R" + A₂ (Aitchison, 1982; Leinster, 2016). Therefore, since tim(in(w)) = d, we also know that im is(ofttmax ow) d-dimensional subspace Ag.
Thus, LLM output probabilities must also reside in a d-dimensional subspace.
Finally, since Ap is an unconventional vector space, it is useful to translate LLM proba- bility distributions into a more conventional vector space. To do so, we use the fact that
Technically, any neural network suffices here.
2 More accurately, the logits will always lie on an al-most-d- diimee-ddimaii subspace. For convenience, we assume full-rank matrices, and thus a d-dimensional subspace.
Our use of "softmax" here corresponds to temperature softmax with temperature 1.
3

---

Published as a conference paper at COLM 2024
R
softmax
log softmax RR))
cir (softmax(R")
Figure 3: Points in the logit space Re (far left) are mapped via the softmax function to points (probability distributions) on the simplex Ap (middle left). Crucially, the softmax maps all points that lie on the same diagonal (shown points of the same color) to the same probability distribution. For numerical stability, these values are often stored as log-probabilities (middle right). The clr transform returns probability distributions to points to a subspace Up of the logit space (far right). The softmax function and cir transform are inverses of one another, and form isomorphism between Up and Ag.
Ag is isomorphic to vector subspace of RD via the softmax function. In particular, Fig- ure 3 illustrates how Ag is isomorphic to the hyperplane Ug that is perpendicular all-ones vector 1p. The isomorphism' s inverse mapping 4a - Up is the center log ratio frans- form cpr(p) log p
the
1 E log Pi. By linearity, im softmax ow) is d-dimensional 2 subspace of Up C R
Thus, LLM outputs occupy d-dimensional subspaces of logit space IR" probability space A., and Up. We call these subspaces the image of the LLM on each given space. A natural consequence this low-dimensionality
that any collection of d linearly independent LLM outputs form basis for the image of the model, i.e., all LLM outputs can be expressed as linear combination these outputs.
3 Obtaining full outputs from API-protected LLMs
As explained in the previous section, in order to obtain the image of an LLM, need to collect outputs (token probabilites for each token in the vocabulary) from the LLM. Unfortunately, most LLM APIs do not return full outputs, likely because full outputs are large and expensive to send over an API, but perhaps also to prevent API abuse, since full LLM outputs contain lots of useful information (Dosovitskiy and Brox, 2016; Morris al., 2023) and can be used to distill models (e.g., Hinton et al., 2015; Hsieh et al., 2023). In their paper, Morris al. (2023) give an algorithm for recovering full outputs from restricted APIs by taking ad advantage of a common API option that allows users to add a bias term 8 S fmax to the logits for specific tokens. The algorithm they describe requires v log(B/e) calls to the API to obtain one full output with precision oog-probabilitis the top-2 tokens. Carlini al. (2024) also propose a logprob-fir method based the same principle.
exactly calls when the API returns
We give a variant their algorithm for APIs that return the log-probability of the top-k tokens that obtains full outputs in v/k API calls. We find that this improved algorithm suffers from numerical instability, and give a numerically stable algorithm that obtains full outputs in // (k- 1) APT calls. We also give practical algorithm for dealing with stochastic APIs that randomly choose outputs from set of π possible outputs, behavior we observe in OpenAl's API. This algorithm allows the collection of full outputs in no/ 2) API calls on average. Table 1 gives an overview of our algorithms with back-of-the envelope cost estimates for specific LLM.
3.1 Full outputs from APIs with logprobs
Our goal is to recover a full-vocabulary next-token distribution E Ap from an API- protected LLM. We will assume that the API accepts prompt on which to condition the

---

Published as a conference paper at COLM 2024
Algorithm
Complexity
API calls per output
Image price (USD)
Logprob-free (Morris et al., 2023)
Allg((mmx// v/k
800000
16338
With logprobs
20000
410
Numerically stable
v/ [k 111-11 1) nv (k - ((d)
25 000
512
Stochastically robust
133000
2724
LLM Image ($4)
800
Table 1:
summary of our proposed algorithms for obtaining full LLM outputs, with estimates for the number of APT calls required per output, and the price of acquiring the model image. Estimates are based on gpt-3. 5-turbbo-liike API LLM with v = 100000,
d = 4096, 10-6 k = 5, fmax 100, and л = 4. Note that the (d) algorithm cannot be used obtain the LLM image, since relies on having LLM image as a preprocessing step.
distribution, as well as a list of up to tokens and a bias term & S fmax to add to the logits of the listed tokens before applying the softmax function. The API returns a record with the k most likely tokens and their probabilities from the biased distribution. For instance, querying the API with maximally biased tokens, which (without loss of generality) we will identify as tokens 1,2,...,k, yields the top-k most probable tokens from the biased distribution p' softmax (f') where
[ 14i+pman iE (1,2,...,k)
(1)
otherwise
and l E IR the LLM's logit output for the given prompt.
Assuming that the logit difference between any two tokens is never greater than fmax, these top-k biased probabilities will be pt, p2...., .K. For each of these biased probabilities p/. we can solve for the unbiased probability
P
p;
(2)
exp Pmax exp Pmax Σ p/
(proof in the SA.1). Thus, for each API call, we can obtain the unbiased probability of k tokens, and obtain the full distribution v/k API calls.
3.2 Numerically stable full outputs from APIs
In practice, the algorithm described in $3.1 suffers from severe numerical instability, which can be attributed to the fast-growing exponential term exp fmax and the term Σ pf which quickly approaches 1. We can eliminate the instability by sacrificing some speed and using a different strategy to solve for the unbiased probabilities. Without loss of generality, let Po be the maximum unbiased token probability. This can be obtained by querying the API once with no bias. we then query the API and apply maximum bias only to tokens 1, k, then the API will yield P/ Py....,, Pk and p/- We can then solve for the unbiased probabilities of the k tokens
Pj exp (log p/- fmax - logp's + loggr)
33)
(proof in SA.2). By finding k- 1 unbiased token probabilities with every API call, we obtain the full output in o/ vk(k - calls total. Algorithm gives a formal description this method.
3.3 Full outputs from stochastic APIs
Each of the above algorithms assume that the APT is deterministic, i.e., the same query will always return the same output. However, this may not always be the case; for instance, we find that OpenAl's LLM APIs are stochastic. While this would seem to doom any attempt at obtaining full outputs from the LLM, it is possible to counter certain types of stochasticity. In particular, model stochastic API behavior collection of outputs p1, ,p,...,,,,

---

Published as a conference paper at COLM 2024
Algorithm Our numerically stable probability extraction algorithm, which takes set of contexts, a set of tokens, and an API for an LLM with vocabulary V and returns the LLM's probabílities for each token in each context. The API takes context and set of token-bias pairs and returns the top-k biased probabilities. P(S) denotes the power set of a set S.
function EXTRACT( contexts S V*, tokens C API: V x P(V x R) - P(V x R) probs E R contexts] x [tokens]
> Initialize empty index to collect probabilities
for E contexts do
(v, Pb) = argmax ( (LP)EAP((CO) P
>> Get top probability token b Partition tokens into batches P Set biases to f for batch tokens P Get biased probabilities
for each batch T S tokens with [T] - do
bias = [i, her
(0..p%) U (1, (1. paer = APl(c, bias)
for E do
probs = exp(log - -
log p/ + +log Po)
Get unbiased probability
end for
end for end for
return probs
end function
from which the API randomly returns from. This might be the result of multiple instances of the LLM being run on different hardware which results in slightly different outputs. Whichever instance the API returns from determines which of the n outputs we get. order to determine which of the outputs the API returned from, let p be the second highest token probability for output p', and observe that log p log p log p log p for all outputs p' and biased outputs p" where tokens a and v - 1 are not biased (proof SA.3). Therefore, by biasing only k - 2 tokens for each call, the API will return p and pー- which can use to find log pt log p which serves unique identifier for the distribution.
Thus, after an average of no/ / (k 2) calls to the API we can collect the full set of probabilities for at least one of the outputs.
4 Fast, full outputs using the LLM image
The dominating factor in runtime for the algorithms in $3 is the vocabulary size v, which can be quite large (e.g., over 100 000 for OpenAl LLMs). We now introduce a preprocessing step that takes advantage of the low-dimensional LLM output space to obtain O(d) versions of all the above algorithms. Since d < v for many modern language models, this modification can result in multiple orders of magnitude speedups, depending the LLM. For instance, the speedup for a model like pythia-708 (Biderman et al., 2023) would be 100x. The key to this algorithm is the observation from $2 that d linearly independent outputs from the API constitute a basis for the whole output space (since the LLM's image has dimension d). pd] E Ag as preprocessing step
We can therefore collect these outputs P [p' p² in ((vd) API calls using any of the above algorithms and d unique prompts, and then use these to reconstruct the full LLM output after only (dd) queries for each subsequent output.
To get new full output p, use any of the above algorithms to obtain P2, Pá- Since resides in d-dimensional space spanned by the columns of P, the rest of the values of are fully determined by these first d values. SB gives the details of how to solve for the remaining values. Thus we can retrieve in only ((d) API queries.
This (v/d)x speedup makes any method that relies on full model outputs significantly cheaper so long as the number of model outputs needed exceeds Such methods include model stealing (Tramèr et al., 2016; Krishna et al., 2019) which attempts to learn model that exactly replicates the behavior of target model, and LM inversion (Morris et al., 2023) which uses LLM outputs to reconstruct hidden prompts. Additionally, the preprocessing

---

Published as a conference paper at COLM 2024
Algorithm 2 Our algorithm for finding the model image and embedding size for API- protected LLM. To get the embedding size of the model without finding the image, extract only the first I token probabilities at each step. With API caching this takes (vvd/K time, or O(d /k) only extract the embedding size.
function GETIMAGE( API: V P(V x R) + P(VX I)) for = 1,2, do
L: = clr (E V. API)) R
Get full logits for contexts When the output rank stops increasing Return the model image. Embedding size is 100.
if rrank (,) f 100 then
return 1,0100 end if end for end function
pythia-70m 512
pythia-160m 768
pythia-410m d 1024
gpt-3. 1-.-3..-tuurr.
4096 d S 4650
106
Singular value
10
10
0
0.5
1.5
0.5
1.5
0
0.5
1
1.5
2
4
6
Index
-103
Index
.103
Index
.103
Index
-103
Figure 4: The singular values of outputs from LLMs with various known and unknown embedding sizes d. For each model with known embedding size, there is clear drop in magnitude at singular value index d, indicating the embedding size of the model. Using this observation, we can guess the embedding size of gpt-3. 5-turbo to be 4096.
step can be computed once then shared between any number of clients, further diluting the cost of full outputs.
5 Discovering the embedding size of API-protected LLMs
Assuming only the generic output layer described in Figure 2, it is possible to infer the embedding size d of an APl-protected LLM from its outputs alone. Since the model outputs occupy d-dimensional subspace of Ar, collecting d linearly independent out- puts ,', p²,p,,..., pd from the LLM forms a basis for the LLM's image. In other words, subsequent model outputs will be a linear combination of the first d outputs. We can there- fore discover the value of d by collecting outputs one time until the number of linearly independent outputs in the collection (i.e., the rank) stops increasing, which will occur after we have collected prompts. We find that it suffices slightly over-collect, e.g., d + 1000 outputs. Algorithm 2 formalizes this procedure.
To find the rank of model outputs, we use the fact that a matrix with linearly independent columns will have d non-zero singular values. We use singular value decomposition to cir(pt 1000)] (100)) and
obtain the singular values of the matrix L [cIr(p)) clr(p2) observe the index at which the magnitude of the singular values drops to zero, which occurs at index d. In practice, numerical imprecision causes the magnitudes drop almost to zero.
To validate our method, we collect next-token distributions (conditioned on unique, 1-token prompts) from several open-source LLMs from the Pythia family (Biderman et al., 2023) with embedding sizes ranging from 512 to 1024. For all these models, the singular values of the resulting output matrix drop precisely at index d, as shown in Figure 4.
7

---

Published as a conference paper at COLM 2024
10
Residual
10a
pythia-70m
2ythia-70m-dedupel
pythia-160m
10-5
0.2
0.4
0.6
0.8
1.2
1.4
Training step
-10
Figure 5: Residuals of the least-squares solution of Lx = l for an output & from the pythia-70m checkpoint at training step 120 000, and output matrices L from various Pythia model checkpoints. High residuals indicate that the output is not in a model's image.
To demonstrate our method's effectiveness against APT-protected LLMs, we use our algo- rithm from $3.3 to collect nearly 6,000 next-token distribution outputs from gpt-3.5-turbo popular APl-protected LLM whose embedding size is not publicly disclosed. We find that this model's singular values drop between index 4,600 and 4,650, indicating that the embedding size of this model is at most this size. This predicted embedding size is some- what abnormal, since LLM embedding sizes are traditionally set to powers two. If this were the case for gpt-3. 5-turbo, 50trrb it would be reasonable to guess that the embedding size is 22 4096. We predict that our raw estimate of 4600-4650 is an overestimate of the true embedding size, since any abnormal outputs due to errors (whether in our own code or Ap')AI's) would only increase the dimensionality of the observed output space. For instance, if we inadvertently collected 504 corrupted outputs, then a model with embedding size 4096 would appear to have an embedding size of 4600.
Knowing the embedding size alone is insufficient to ascertain LLM's parameter count. However, since most known transformer-based LLMs with embedding size 4096 have around 7 billion parameters, it is likely that gpt-3. 5-turbo has similar number of active parameters. Any other parameter count would result in either abnormally narrow or wide models, which often perform worse. The actual parameter count may be much higher if the model uses a tnixture-of-expprtt. architecture (Shazeer et al., 2017). Given the active development and decreasing cost of inference with gpt- 3.5-turbo, it is possible that its size and architecture has changed over time. Fortunately, our method can be used to monitor these updates over time, alerting end-users when LLM providers change embedding size (and presumably therefore, model size).
Attributing model outputs and auditing model updates
An LLM image can serve as identifying signature for the model, shown in Figure where the logit output from a pythia-700 70m checkpoint lies uniquely in the checkpoint's image, and not the image of the preceding or following checkpoints, nor the checkpoints of any other similar model.5 This suggests that LLM images are highly sensitive to parameter changes, which makes sense because the intersection of two different d-dimensional spaces an even lower-dimensional space. Thus, it is possible to determine precisely which LLM produced particular full-vocabulary output using only API access to set of LLMs, and notably, without knowing the exact inputs to the model.
This finding has implications for LLM auditing (Mökander et al., 2023), such obtaining granular information about updates to APT-protected LLMs. For instance, if the LLM image remains the same but the logit outputs change, this would indicate partial model update where some part of the model changes but the softmax matrix remains the same. This might happen when the LLM has hidden prefix added to all prompts and this hidden prefix changes, or when some part of the model was updated while leaving the softmax
We specifically use model version 0125, accessed February 1-19,200 2024.
5Open source models with detailed checkpoint information (e.g., Biderman et al., 2023; Liu et al., 2023) make this type of detailed analysis possible.
8

---

Published as a conference paper at COLM 2024
Change
Interpretation
No logit change, no image change Logit change, no image change Low-rank image change (57.1)
No update
Hidden prompt change or partial finetune
LoRA update
Image change
Full finetune
Table 2: Implications of image/logit changes.
matrix unchanged. Table 2 gives an overview for how to interpret various combinations of detectable API changes.
Using the same principle, one can also detect malicious use of open-source LLMs. If provider attempts to profit off of an open-source model with non-commercial license, auditor can reveal the scheme by checking the LLM image, even the provider used hidden prompt to mask the model identity. It would be unlikely for images of two different LLMs to match, unless purposely designed to do so. Note that this test is one-sided, and malicious providers may slightly fine-tune LLM to avoid detection.
More Applications
Access to the LLM's image unlocks several additional capabilites, some of which we review below. We leave further investigation of these methods for future work.
7.1 Detecting LoRA updates
Access to the LLM image can afford even finer granularity insight into LLM updates. For instance, as LoRA (Hu et al., 2022) is popular parameter-efficient fine-tuning method which adjusts model weights with low-rank update AB where A R and B E R so that the softmax matrix W E Rexd becomes W + AB. may be possible detect these types of updates by collecting LLM outputs before (L E Roxd) and after (L' E Roxd) the update and decomposing them as WH = L and (W + AB) = L' where H, H' E Raxd such decomposition is found then is likely that the weights received a low-rank update. We leave it to future work to find an efficient algorithm for this decomposition.
7.2 Finding unargmaxable tokens
Due to the low-rank constraints on LLM outputs, it is possible that some tokens become unargmaxable (Demeter et al., 2020; Grivas et al., 2022), i.e., the token always has less probability than some other token. This happens when the LLM's embedding representation of the token lies within the convex hull of the other tokens' embeddings. Previously, finding unargmaxable tokens appeared to require full access to the softmax matrix W, however, it is possible to identify unargmaxable tokens using only the LLM's image, which our method is able to recover. This technique allows API clients to find tokens that the model is unable to output and thereby elicit unexpected model behavior.
7.3 Recovering the softmax matrix from outputs
One might the image to approximately reconstruct the output layer parameters. We hypothesize that LLM embeddings generally lie near the surface of a hypersphere in Rd with small radius r. We see evidence of this in the fact that the Pythia LLM embedding norms are all small and roughly normally distributed, as shown in Figure 6 in the Appendix. We can attempt to recover W up to rotation by assuming that all embeddings must have unit magnitude, then, given a matrix L E Ro of model logits, we can find W (up to rotation) by finding a decomposition WH L such that for all || This solution may also be approximated by finding the singular value decomposition WEV of L, though rows of this W will have magnitude less than 1.
9

---

Published as a conference paper at COLM 2024
7.4 Basis-aware sampling
In a recent paper, Finlayson et al. (2024) propose a decoding algorithm that avoids type-I sampling errors by identifying tokens that must have non-zero true probability. Importantly, this method relies on knowing the basis of the LLM's output space, and is therefore only available for LLMs whose image is known. Our approach for finding the model image makes this decoding algorithm possible for API LLMs.
8 Mitigations
We consider three proposals that LLM providers may take to guard against our methods. The first proposal is to limit or entirely remove API access logprobs. This is theoretically insufficient, as Morris et al. (2023) show that it is possible to obtain full outputs using only information about the biased argmax token, albeit inefficiently in vlog(B/e) API calls. Providers may rely on the extreme inefficiency of the algorithm to protect the LLM, as OpenAl appeared to do in response to Carlini et al. (2024). However, our algorithm in $4 brings the cost down to feasible Od log (B/ ee) API calls per output once the initial work of finding the LLM image finishes, the result of which can be made public and reused.
The second proposal is to remove API access to logit bias. This would be effective, since there are no known methods to recover full outputs from such an API. However, logit bias has several useful applications, such as for blocking undesirable tokens and controlling text generation, and making such a restriction could seriously degrade the usefulness the API.
Lastly, we consider alternative LLM architectures that do not suffer from softmax bottle- neck. There are several such proposed architectures with good performance (e.g,, Xue et al., 2021; Yu et al., 2023; Wang et al., 2024). Though this is the most expensive of the proposed defenses, due to the requirement of training a new LLM, it would have the beneficial side effect of also treating other tokenization issues that plague harge-vocabular LLMs (e.g., Itzhak and Levy, 2022). A transition to noftmax-bottleneck-ee free LLMs would fully prevent our attack, since the model's image would be the full output space.
Conclusion
We have shown how the low-rank constraints imposed by the softmax bottleneck expose non-public information about APl-protected LLMs. We also demonstrated how this infor- mation can be used to efficiently extract full model outputs, expose model hyperparameters, function as a unique model signature for auditing purposes, and even detect specific types of model updates, among other uses. We find that some current protections against these attacks are insufficient, and that the more effective guards tend to inhibit sanctioned API use cases (i.e., logit bias), or are expensive to implement (i.e., changing model architecture).
Overall, we find that the benefits of our proposed methods outweigh the harms to LLM providers. For instance, allowing LLM APÍ users to detect model changes builds trust between LLM providers and their customers, and leads to greater accountability and transparency for the providers. Our method can also be used to implement efficient protocols for model auditing without exposing the model weights. On the other hand, discovering the embedding size of the LLM does not enable a competitor to fully recover the parameters of the LLM's softmax matrix or boost performance of their own model. Even efficiently extracting full outputs for model stealing or inversion (54) is unlikely to have detrimental effects, these are already known threats and are only made more efficient by our methods. We therefore believe that our proposed methods and findings do not necessitate a change in LLM API best practices, but rather expand the tools available to API customers, while informing LLM providers of what information their APIs expose.
References
J. Aitchison. The statistical analysis of compositional data. Journal of the Royal Statis- fical Society: Series B (Methodological), 44(2):39--100, 1982. doi: https://doo..rg////
10

---

Published as a conference paper at COLM 2024
18. 1111//2517-66611182.000019999999 URL 1982. (bb1199.x.. 2
Markus Anderljung, Everett Thortton Smith, Joe O'Brien, Lisa Soder, Benjamin Bucknall, Emma Bluemke, Jonas Schuett, Robert Trager, Lacey Strahm, and Rumman Chowdhury. Towards publicly accountable frontier Ilms: Building external scrutiny ecosystem under the aspire framework, 2023. URL hwtph: www.://rg/ab/////e/eee 14711.
Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pages 2397- 2430. PMLR, 2023. 4, 5,5
Nicholas Carlini, Daniel Paleka, Krishnamurthy Dj Dvijotham, Thomas Steinke,Jonathan Hayase, A. Feder Cooper, Katherine Lee, Matthew Jagielski, Milad Nasr, Arthur Conmy, Eric Wallace, David Rolnick, and Florian Tramèr. Stealing part of a production language model, 2024. 1, 1,3,8 8
David Demeter, Gregory Kimmel, and Doug Downey. Stolen probability: A structural weak- ness of neural language models. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault, editors, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2191-2197, Online, July 2020. Association for Computational Linguistics. doi: 10. 10.18653/ve /ttp:///mm/0a.ammmmma...999 URL https: /ccl https////aaaaaallooaooooo wwt. acl-main. 198. 7.2
Alexey Dosovitskiy and Thomas Brox. Inverting visual representations with convolutional networks. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4829-4837, 2016. doi: (0.1109/CVVR..006.5222
Matthew Finlayson, John Hewitt, Alexander Koller, Swabha Swayamdipta, and Ashish Sabharwal. Closing the curious case of neural text degeneration. In The Twelfth International Conference on Learning Representations, 2024. URL
net net/forum?id
dONpC9GL10. 7.4
Andreas Grivas, Nikolay Bogoychev, and Adam Lopez. Low-rank softmax can have unargmaxable classes in theory but rarely in practice. In Proceedings of the 60th An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6738-6758, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/ http:////1122222alllooo
URLhttps://aclam
org 2ttp acl- 465. 7.2
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. In Proc. of NeurlPS, 2015. URL http:::/arkkiv.rr//bb//////// 02531.
Cheng-Ya Yu Hsieh, Chun-Liang Li, Chih-kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alex Ratner, Ranjay Krishna, Chen-Yu Lee, and Tomas Pfister. Distilling step-by-step! out- performing larger language models with less training data and smaller model sizes. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Findings of the Association for Computational Linguistics: ACL 2023, pages 8003-8017, Toronto, Canada, July 2023, Association for Computational Linguistics. doi: 10.18653/v1 /220://......../ggg-.m.. acl 507.. URL 2wtp:t//hho.oooooo//22.. org/2023. 11122.ffinding findings-ach meddings-acl.500 587.
Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL forum?id=nZZeWeeeFyyy 7.1
net/
Itay Itzhak and Omer Levy. Models in a spelling bee: Language models implicitly learn the character composition of tokens. In Marine Carpuat, Marie-Catherine de Marneffe, and Ivan Vladimir Meza Ruiz, editors, Proceedings of the 2022 Conference of the North Ameri- can Chapter of the Association for Computational Linguistics: Human I Language Technologies,
11

---

Published as a conference paper at COLM 2024
pages 5061-5068, Seattle, United States, July 2022. Association for Computational Lin- guistics. doi: naacl-main. 373. 8
URL https://aclaattholgyy org/2022.
Kalpesh Krishna, Gaurav Singh Tomar, Ankur P. Parikh, Nicolas Papernot, and Mohit lyyer.
Thieves on sesame street! model extraction of bert-based apis. ArXio, Abss/1910.12550 2019. 4
Tom Leinster. How the simplex is a vector space. https://golent ph. ph. utexas wwlessseddcceeeeeeee adu/category/ httph//wwwtteeseeeeeessseeeeeeee
html, 2016. Accessed: 2024-03-12.
Zhengzhong Liu, Aurick Qiao, Willie Neiswanger, Hongyi Wang, Bowen Tan, Tianhua Tao, Junbo Li, Yuqi Wang, Suqi Sun, Omkar Pangarkar, Richard Fan, Yi Gu, Victor Miller, Yonghao Zhuang, Guowei He, Haonan Li, Fajri Koto, Liping Tang, Nikhil Ranjan, Zhiqiang Shen, Xuguang Ren, Roberto Iriondo, Cun Mu, Zhiting Hu, Mark Schulze, Preslav Nakov, Tim Baldwin, and Eric Xing. Llm360: Towards fully transparent open- source lims, 2023. 5
John X. Morris, Wenting Zhao, Justin T Chiu, Vitaly Shmatikov, and Alexander M. Rush. Language model inversion. ArXio, abs/ ftss/2311.1364,, 2023. 3, 4,
Jakob Mökander, Jonas Schuett, Hannah Rose Kirk, and Luciano Floridi. Auditing large language models: a three-layered approach. Al and Ethics, May 2023. ISSN 2730-5961. doi: 1,
(0..0077/436611-23--00299 URL http://dd..oi..rrg/1000 1ttp:///..000//4881-002-------00
OpenAl, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, llge Akkaya, Flo- rencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, al. Gpt-4 technical report, 2024. 1
Noam Shazeer, * AAaaia Mirhoseini, *Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gates mixture- of-experts layer. In International Conference on Learning Represenlations, 2017. URL https: //openreview. net mett/oouumiddBB Begrrrn ckMDqlg. 5
Florian Tramèr, Fan Zhang, Ari Juels, Michael K. Reiter, and Thomas Ristenpart. Stealing machine learning models via prediction apis. In USENIX Security Symposium, 2016. URL https://api semanti cscholarr orggCooppSID:298AA 2984526. 4
Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Neural In- formation Processing Systems, 2017. URL https: semanticscoolar org//orpusII: 13756489. 2
Junxiong Wang, Tushaar Gangavarapu, Jing Nathan Yan, and Alexander M. Rush. Mam- babyte: Token-free selective state space model. ArXiv,
2024. 8
Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel. Byt5: Towards token-free future with pre-trained byte-to-byte models. Transactions of the Association for Computational Linguistics, 10:291-306, 2021. 8
Zhilin Yang, Zihang Dai, Ruslan Salakhutdinov, and William W. Cohen. Breaking the softmax bottleneck: A high-rank RNN language model. In ICLR, 2018. URL https: ///pppreeieee.ee/foroutittttieee www..e///oorumiicckwwZ------Z 1,
Lili Yu, Daniel Simig, Colin Flaherty, Armen Aghajanyan, Luke Zettlemoyer, and Mike Lewis. Megabyte: Predicting million-byte sequences with multiscale transformers. In A. Oh, T. Neumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Ad- vances in Neural Information Processing Systems, volume 36, pages 78808-78823. Curran Associates, Inc,, 2023. URL https: http://pccccccc.ceeeecpcoo ieeerripsaae cccp/pper_illls/ppperer//e//e//e ftl.ss//ppper//22////// file//88778f8043f338880 34
pdf. 8
12

---

Published as a conference paper at COLM 2024
A
Proofs
This appendix contains derivations for the equations used to solve for unbiased token probabilities given biased LLM outputs.
A.1 Fast logprobs algorithm
Our goal is to prove
pj exp fmax exp mmxx Lj= I1
(4)
Σ P)
Proof. We begin with the definition of the softmax function
exp(e; + (max)
p1
(5)
1 exp(L, + fmax) + Ej-k-1 exp
We then rearrange to obtain
exp exp(1; f; + Fmax)
Σ
Σ exp(ly + fmax),
(6)
P/
j=++1
the left hand side of which is independent of the bias term, meaning it equivalent when bmax = 0, i.e.,
exp &
exp(e, + Bmax)
Σ exply ej
Σ exp(le + Pmax).
Pi
P/.
We can rearrange
exp &
exp(l; + Bmax)
k
Σ exp(1, (f, + maax) + E exply
(8)
pi
p/
|=1
exp lj
Pi explfi+tPmax)
(9)
E/=1 exp(lj + Bmax) + E/=1 expl
71
and expand
p/exp6 e
pi =
(10)
exp( + fmax)
2/ exp(li (e, + fmax) + p/ E exp lj
pf² exp( (ap(-Bmax) exp(e: + Bmax)
(11)
exp((; + fmax) p 1/1 exp(ep (l, + mmax) + p\, exp((pmmmxm Pmax) Ej=1
exp(1, fmax)
and finally simplify by multiplying the top and bottom of the right hand side by
(12)
Σ exp(l; + Bmax) + Ej=k+1 exp lj
and which converts each term of the form exp(f; + bmax) to p/, resulting in
p.? exp( (app--mmxx
Pi
(13)
P p - P/ + pxppp(
Tepp--mman Bmax 2j=1 1\}
p.. eleppppFpp (-Bmax)
(14)
1 - p/ + exp( -Bmax) x 1/
P/
(15)
exp Bmax exp Bmax
P//
which concludes the proof.
13

---

Published as a conference paper at COLM 2024
A.2 Numerically stable algorithm
The next proof is much simpler. Our goal is to prove
pj exp pi
Bmax log po log Po).
(16)
Proof. We begin with four facts
exp Lj P: Ej exp fj
exply ej - expp ej
exple bo Ej- exp Ej
exp lo
P
pv
Pb
Dj exp ej
(17)
which follow from the definition of softmax. Combining these, we have
Pi
Po exp Ep
pl.
P'b exp lu
and
(18)
exp
exp e
Combining these again, we get
p/ p'eep p!!
Pi
(19)
Po exp 2
which we can rearrange obtain
Pbp: expl, L P/P
(20)
exp e
Next, we use the fact that exp P exp maxx exp to get
exp(- fmax)_
(21)
ppP
which we can take the log of, rearrange, and exponentiate achieve our goal
pi = exp(log p\ Bmax - log pb + log pv).
(22)
A.3 Stochastically robust algorithm
We would like to derive
log pr. log Po 1 = log pb log P'
(23)
Proof. Using our result from SA.2, we have
p; = expplog/) p fmax - log + log pv)
(24)
p; = exp(log p fmax log + log 1)-1)
(25)
Simply setting the right hand sides equal to one another, taking the log of both sides, then subtracting identical terms from both sides gives us our goal.
B Solving for full outputs using the LLM image
Given a matrix P E Ag whose columns span the LLM image, and Pı, P2...., Pá, we can solve for p E dg. To do this, we will use the additive log ratio (alr) transform, which is an isomorphism A, R and defined
alr(p) = (1og P2
P3
Pe
log
log
(26)
p1
14

---

Published as a conference paper at COLM 2024
Frequency
0.5
2.5
3
3.5
4.5
5
5.5
6.5
Embedding norm (L2)
Figure 6: Softmax matrix row magnitudes (here from pythia-70m) are small and are dis- tributed approximately normally within narrow range.
to transform the columns of and into vectors in R though since we only know the first d values of p, we can only obtain the first d values of alr(p). Because the air transform is an isomorphism, we have that the columns of
air alr(P) = [alr(p')
alr(p²)
alr(p")] E R(v 1)xd
(27)
form basis for d-dimensional vector subspace of R and alr(p) lies within this sub- space. Therefore, there some x R² such that alr(P)x alr(p). To solve for x, all that is required is to find the unique solution to the first d rows of this system of linear equations
aal((p)) ) alr(p') )2 alr(p²)2
alr(p²)
alr(p*), ) air p 2
x1 X2
aal p)1
aIr alr(p)2
(28)
alr(p")a )a alr(p²)a
air(pa) )a
[xa
a
After finding x, can reconstruct the full LLM output p air (alr( where the inverse function is defined as
alr (x) 1 Σ expx
ex, x1, exp X2. exp X. xp-1)-
(29)
15


Reviews:
[{"review_id": "4jwYdS1y9i", "reviewer": "Reviewer_D4ia", "summary": "This paper explores the vulnerabilities in the API deployment of large language models, specifically focusing on how a limited number of API queries can expose non-public information about such models. The authors demonstrate that softmax bottleneck deduces extensive details about a model's structure, such as its embedding size. The study further suggests methods for LLM providers to shield against these vulnerabilities, while also proposing that these weaknesses could enhance transparency and accountability in LLM operations.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We value your input and hope we can address your concerns:\n\nWe agree that the attack is defendable, but assert that exposing it is still valuable because many LLM APIs were vulnerable to it. Otherwise, the unknown vulnerabilities would persist. As described in Sec. 8, the defenses themselves have drawbacks, which is likely why OpenAI’s recent patch kept logit bias. \n\n> Outcomes/purpose of attack. Harmful? \n\nOur attack reveals the LM image, which has several applications listed in the Secs. 4-7. We agree (last par., Sec. 9) that the \"attack\" is not egregiously harmful to LLM providers (though OpenAI likely hides model details for a reason). Providers may opt to *keep* these \"vulnerabilities\" to build trust, given the useful, benign capabilities we propose (e.g., detecting model updates.)\n\n> Distinction from [1]\n\nOur **overlapping** contributions with [1] include\n- Algorithm for recovering logits.\n- Theory / algorithm for recovering the output layer (the “image”).\n- Finding the embed size.\n\nNotably, [1] claims \"no immediate practical consequences\" of the attack, but we show several (Secs. 4, 6, 7). Our **unique** contributions include\n- Concrete estimate of OpenAI model size (5)\n- Fast logit extraction (4)\n- Model output attribution, update detection (6)\n- List of additional applications (7)\n\nWe will outline these more explicitly in a “Contributions” section. We hope our work will be judged for both our **overlapping and unique contributions**, since we released it contemporaneously with [1]. Please note that OpenAI already patched their API when [1] came out, so we could not have conducted this research after [1]’s release.\n\n> Recovering last layer\n\nWe would like to clarify that our “model image” corresponds to the “last layer” recovered in [1], both of which are linear transformations of the parameters in the last layer. We will make this explicit.\n\n> Suggested application (Sec. 7).\n\nWe use Sec. 7 as “future work” to show that there are many applications (beyond the paper’s scope) aside from those detailed in Secs. 4-6. We will edit to convey this clearly.\n\n> Countermeasure experiments\n\nWe cite (first par., Sec. 8) Morris et al. (2023) who already demonstrate logprob-free logit extraction. Other countermeasures block our algorithm, so it is unclear what experiment to run.\n\n> Theory, overview of algo.\n\nThank you, we will more clearly highlight our algorithms in our edits. Do you have specific feedback on how we could make the theory more comprehensive?", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "+ The findings are interesting, albeit it is a concurrent work to [1]. \n\n+ The techniques for \n\n+ There are suggestions for: (1) employing this method for model auditing, and (2) mitigating this type of attack.\n\n[1] Stealing part of a production language model, published on arXin in March 11th.", "reasons_to_reject": "+ The attack can be easily defended by blocking logits or adding noise to the outputs. \n\n+ Although several types of attack has be discussed in this work, but the main outcomes and purpose of this attack is unclear. \nHow these findings can be orchestrated as a comprehensive attack that harms LLM applications. I would consider discovering size of h in LLM is minor as it is just a number and we still need to train the models. Recovering the exact logits of the top K tokens is still minor given the 'biased' top K logits already provides essential information.\n\n+ Improvements are needed in the clarity and organization of the writing:\n  + A comprehensive overview of the algorithm and underlying theory would enhance understanding.\n  + Clarification is needed on how this work differs from [1]. Specifically, if Sections 7 and 8 are intended to address these differences, a more thorough analysis in these sections could help establish a clear distinction from prior work\n\n+ The suggested applications (in Section 7) lack sufficient detail and are not adequately supported by theoretical frameworks or experimental validation.\n\n+ The countermeasures suggested are not substantiated by experimental evidence.\n\n+ The paper was submitted several weeks after [1], (and arXiv version is also several days after [1]) yet it does not adequately distinguish its theoretical contributions or empirical analysis from those found in [1]. For example, an algorithm to recover the last layer was proposed in [1] but not discussed in this work. I would consider recovering the 'shape' is far less interesting than recovering 'parameters' of the linear module. This means the missing piece to [1] is the most essential part of this work. \n\nI am aware that this is a concurrent work to [1], and my evaluation is based on the quality of this paper.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jcPppcA2mW", "reviewer": "Reviewer_CQvU", "summary": "This paper discussed an interesting method to partially recover some important meta data of black box LLM, primarily the hidden dimension before embedding matrix. \n\nThe core idea is that the output logit vectors reside in dimension $d$ which is much smaller than vocabulary size $V$, thus a singular value decomposition over a collection of inferred logits matrix successfully recovers the actual hidden dimension.\n\nThe authors further discussed extensions such as inferring the model that generates certain outputs or auditing for any model updates, etc.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your helpful review and insights. \n\n> Comparison with Carlini et al. (2024) [1]\n\nWe hope our work will be judged for both our **overlapping and unique contributions**, and not simply for the additional value, since we released it contemporaneously with [1]. Please note that OpenAI already patched their API when [1] came out, so we could not have conducted this research after [1]’s release. We will outline these more explicitly under a “Contributions” section. \n\nOur **overlapping** contributions with [1] include\n- Algorithm for recovering logits.\n- Theory / algorithm for recovering the output layer (the “image”).\n- Finding the embed size.\n\nNotably, [1] claims \"no immediate practical consequences\" of the attack, but we show several (Secs. 4, 6, 7). Our **unique** contributions include\n- Concrete estimate of OpenAI model size (5)\n- Fast logit extraction (4)\n- Model output attribution, update detection (6)\n- List of additional applications (7)\n\n\n> Primary breakthrough\n\nThe primary breakthrough of our paper is our algorithm for finding the LM image (overlapping with [1]) and a collection of applications of the model image (not discussed in [1]).\n\nOne of these applications is the “fast logit extraction” algorithm we present in Sec. 4, which is our unique contribution. Comparing it to our logit extraction algorithms in Sec. 3 (which are equivalent to those used in [1]) we achieve a speedup from $O(v)$ API calls to $O(d)$ API calls. Note that this new algorithm can only be used *after* the model image is found, i.e., it is not useful for finding the model image. We compare the complexity of these algorithms in Table 1. We will make these points clearer in the paper.\n\n> LoRA updates\n\nThank you for the pointers for section 7. We agree the LoRA update idea is speculative and will revise the language to make this clearer as potential future work. Our other suggested applications (unargmaxable tokens, basis-aware sampling) are theoretically sound and well developed in their respective papers, since both algorithms take the model image as an input. The proposal in Sec. 7.3, recovering the softmax matrix, is a natural extension of finding the model image (this is also explored in [1], though their “recovered matrix” is equivalent to what we call the “model image”.)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Though the core idea of this paper appears to be similar to [Carlini 2024], I think following are reasons to accept this paper:\n\n1. This paper made a clearer algorithms and their complexities under different assumptions (such as w/ logprobs, numerically stable, scholastically robust, etc.)\n\n2. This paper opened up more discussions on impact of the discovery. Including applications on detecting for any backend model change, detecting the source of the LLM mode, and ways to mitigate the stealing of meta information, etc.\n\n3. I appreciate the authors revealed the actual number of estimated hidden dimensions of GPT3.5 (as oppose to [Carlini 2024], which is hidden from public).", "reasons_to_reject": "The primary concern on this paper is the contribution on top of [Carlini 2024]. After comparing with two papers I feel like authors need to make it clearer on what additional value this paper brings to research community. Right now the authors did not have such comparison thoroughly. Thus, I would like the authors to include a \"Contribution\" section to clarify this.\n\nIs the main breakthrough of this paper lies in the algorithmic complexity of doing LLM invocations? If so, I expect to make it clear by comparing big Os of this paper vs. [Carlini 24].\n\nThough I appreciate the detailed discussions on potential future applications, I doubt the feasibility of quite a few. For instance \"detecting the LoRA updates\", the reality is people could simply skip lora updates on embedding layer; and the wording \"It may be possible to detect these types of updates\" sounds sneaky. It should be avoided in academia papers.", "questions_to_authors": "My questions are mostly tied to \"reasons to reject\" section. Please respond to that section.", "extra_scores": {}}, {"review_id": "0ryZd7OBvh", "reviewer": "Reviewer_gbAq", "summary": "This paper presents an algorithm to extract proprietary LLM information from an API-protected black-box LLM.\n\nThe algorithm uses the observation that LLM outputs are \"softmax-bottlenecked\", or they occupy only a low-dimensional subspace of the `R^vocab_size` output space. Using this observation, along with logit bias access to APIs, the algorithm can efficiently extract this low-dimensional subspace. This subspace can be leveraged to 1) infer the hidden dimension of the softmax layer; 2) extract a fingerprint of the API, which can be used to monitor API updates (like LLM changes).\n\nThe algorithm is tested in a real-world setting on the OpenAI GPT3.5 Turbo model, and is shown to work with high accuracy.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive review and for noting that our findings affect millions of LLM users and could lead to better transparency and accountability for API providers. \n\n> Logit bias could go away\n\nWe agree that logit bias may not be a permanent feature, and some APIs do not offer it at all. Our work remains relevant because it deals with common real-world APIs that would likely continue to have this vulnerability otherwise. Alternatively, if logit bias is useful for model auditing, it could become common practice to expose this feature as part of responsible AI deployment.\n\n> Limited information leakage, robustness of model signature for accountability\n\nThank you for this thoughtful discussion! As this a new area of research, it is indeed an open question just how far this type of attack can go. As established in our paper, these attacks may be relatively easy to counter, but they are nonetheless important considerations when building LLM interfaces now and in the future. While we agree that our paper is mostly concerned with details about the LLM final layer, we believe this is an important first step. Indeed, we are working on a followup that gleans information by looking at deeper LLM layers. \n\n> Other models\n\nFor open-source models, we found it sufficient to use Pythia models as generic stand-ins since there is no theoretical or practical reason the attack would fail with, say, GPT2. For closed-source models, we found GPT-4 turbo to be more expensive to attack, and smaller models would be of less interest. Moreover, the experiments would be considerably more expensive to run now since OpenAI has patched their API.\n\n> Post-processing outputs\n\nThe watermarking method in [1] would not change these logprobs, since it affects sampling, not logprobs. Changing temperature, so long as the temperature is constant for each query, should not affect the image, since this simply scales the logit vector by a constant.\n\n> Routing models, mixture of softmax\n\nWe found that, for the same query, the logprobs reported by the API were randomly selected from a set of values, leading us to develop the “stochastic logit extraction” method (Sec. 3.3). Some of the cases you mention are handled by this method. It comes down to whether each query’s output passes through the same softmax matrix. In the case of mixture-of-softmax and other nonlinear output layers, our current algorithm would not work.\n\nThank you for your typos and suggestions. We will fix/add them.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper presents technically solid work, with a very creative algorithm to extract proprietary information with high accuracy from a black-box LLM.\n\n2. The presented attack is on a real-world LLM used by millions of people (the OpenAI GPT3.5 Turbo model), and not just in toy settings. This is quite interesting and impressive, given how secretive OpenAI has been about their model's internals. Moreover, the proposed algorithm is economically feasible (< 1000$ API credits).\n\n3. This paper is very well-written, with a good mix of low-level linear algebra details of the algorithm, and high-level discussions on the implications of the attack.\n\n4. The biggest implication of the method (in my opinion) is forcing more transparency from API providers on silent backend changes. Section 6 shows that even consecutive checkpoints can have very different images, which means that extracted fingerprints are fairly sensitive, and can be used to monitor API updates.", "reasons_to_reject": "This is a great paper overall and a good fit for COLM. I have a few concerns, but overall I think the strengths outweigh the weaknesses.\n\n1. The method (as well as prior works), rely on API providers allowing users to adjust logit biases. Based on Section 8, this method will not work without logit biases. However, I think logit bias is a fairly limited hack from OpenAI's perspective, rather than a long-term solution for controlled decoding. With rapidly improving instruction following capabilities, I think models maybe able to achieve effects similar to logit bias with just natural language instructions in the prompt, and get more natural-looking output. For reference, I don't see logit bias access to the Claude API (https://docs.anthropic.com/claude/reference/messages_post) or the Gemini API (https://ai.google.dev/api/python/google/generativeai/GenerationConfig).\n\n2. Overall, I think the amount of information leakage from the attack is fairly limited, which limits the implications of the work. The only major information extracted is the dimensionality of the softmax matmul layer, which leaves an auditor speculating about the rest of the giant LLM (the more important proprietary information). The signature / fingerprint aspect of the attack is definitely interesting, but Section 6 and Table 2 makes me think there are too many variables that could change the API signature. This could include innocuous changes like timestamp in the hidden prompt, or search results in the hidden prompt of a RAG-enabled API. Watermarking, routed models, prompt preprocessing, output post-processors (safety filters?) could also change signatures (see questions below). Given that the APIs are rapidly changing due to industrial competition, frequent changes in API signature could leave auditors confused.\n\n3. (minor, likely an unreasonable request) I was a bit surprised that experiments were only done on one proprietary model, and one family of open-source models. In particular, were there some cost bottlenecks to run the experiment on GPT-4 Turbo models?", "questions_to_authors": "Questions:\n\n- What happens if the logits are postprocessed before being sent to the softmax? In particular, I think the addition of a watermark [1] could change the image of the LLM without changing the underlying LLM model. I also have a similar question for temperature changes in the API.\n\n- How well does the method work with APIs which are not a single LLM (they route queries between two or more LLMs, like [2, 3, 4]). Does that muddy the image procured by this method, possibly adding to the estimated \"effective dimension\"? What if routing weights are changed? Similar question for the mixture-of-softmax method from [5].\n\nTypos / presentation:\n\n- The paper is missing a related work section, and I think more references would help in better grounding the work. Notably, the work is closely related to hyperparameter extraction and model extraction efforts (some examples: [6, 7, 8, 9]).\n\n- page 3: \"we turn out attention\" --> \"we turn our attention\"\n\n- page 3/ 4: Figure 3 is confusing and I don't think the translation to the clr space is helping my understanding (possibly better for the Appendix). Please correct me if I'm wrong, but my understanding reading the text is that everything after the softmax matmul matrix (logits, softmax, log-softmax) are in a d-dimensional subspace of R^v. Figure 3 feel like there is further reduction in image dimensionality from logits --> softmax-out (which I don't think is true).\n\nHere's an alternative suggestion: Present a table with three columns being (transformer operation, vector space dimension, effective image dimension). Something like the table below:\n\n(transformer operation, vector space dimension, image dimension)  \n(end of transformer sub-blocks, R^d, R^d),  \n(softmax MatMul `l = Wh`, R^V, R^d),  \n(softmax, `p = e^l / sum(...)`, R^V, R^d),  \n(log-softmax, `q = log(p)`, R^V, R^d)  \netc.\n\nAccompany this table with a simulation / points in a space where V = 3, d = 2, and one figure accompanying each of the rows above. The first row could be a 2-d plot, and all others 3-D plots with the points in a 2-d surface.\n\n[1] - https://arxiv.org/abs/2301.10226  \n[2] - https://arxiv.org/pdf/2310.12963  \n[3] - https://arxiv.org/abs/2207.07061  \n[4] - https://arxiv.org/abs/2211.17192  \n[5] - https://arxiv.org/pdf/1711.03953  \n[6] - https://arxiv.org/abs/1711.01768  \n[7] - https://arxiv.org/abs/1802.05351  \n[8] - https://arxiv.org/abs/1910.12366  \n[9] - https://arxiv.org/abs/2305.15717", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper studies how the logits of a black-box LLM (such as OpenAI’s models) can reveal non-public information about the model. The key idea is that most modern LLMs’ outputs are limited to a low-dimensional subspace of the full output space. Combined with this paper’s another contribution on an improved algorithm to recover the full probability distribution with limited API output, it provides a very efficient way to recover the softmax layer and the rank of it, which consequently leads to various non-public information about the model such as the hidden dimension of the model (which could hint the model size) and the nature of the model updates (e.g., whether or not the softmax layer was updated).

Reviewers acknowledged that the paper is technically solid and made a clear algorithmic contribution under various assumptions such as the availability of logprobs, stochastic changes, etc  (Reviewer CQvU, gbAq). Reviewers also think the paper opens up a broader discussion on forcing more transparency from API providers on silent backend changes, employing this method for model auditing, and developing mitigation for this type of attacks (Reviewer CQvU, gbAq). Reviewers also think the findings are interesting and the revealed information about OpenAI models is valuable (Reviewer CQvU, gbAq, D4ia).

Some concerns have been raised regarding the practical implications of the findings, given that the attacks are defensible, the information it leaks are limited (Reviewer gbAq, D4ia), and it relies on specific assumptions like the presence of logit bias features (Reviewer gbAq).

[At least one review was discounted during the decision process due to quality]
Decision: Accept