Logits of API-Protected LLMs Leak Proprietary Information

Authors: matthew  finlayson
xiang  ren
swabha  swayamdipta
Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-22 20:00:05

Abstract: Large language model (LLM) providers often hide the architectural details and parameters of their proprietary models by restricting public access to a limited API. In this work we show that, with only a conservative assumption about the model architecture, it is possible to learn a surprisingly large amount of non-public information about an API-protected LLM from a relatively small number of API queries (e.g., costing under $1000 USD for OpenAI’s gpt-3.5-turbo). Our findings are centered on one key observation: most modern LLMs suffer from a softmax bottleneck, which restricts the model outputs to a linear subspace of the full output space. We exploit this fact to unlock several capabilities, including (but not limited to) obtaining cheap full-vocabulary outputs, auditing for specific types of model updates, identifying the source LLM given a single full LLM output, and even efficiently discovering the LLM’s hidden size. Our empirical investigations show the effectiveness of our methods, which allow us to estimate the embedding size of OpenAI’s gpt-3.5-turbo to be about 4096. Lastly, we discuss ways that LLM providers can guard against these attacks, as well as how these capabilities can be viewed as a feature (rather than a bug) by allowing for greater transparency and accountability.

Main paper: 
 introduction: 
 As large language models (LLMs) become more capable and valuable, many LLM providers have shifted toward training closed-source production LLMs, accessible only via (paywallrestricted) APIs to protect trade secrets (e.g., OpenAI et al., 2024) . As a useful feature, these APIs often allow users to access the probabilities (or logits, i.e., unnormalized scores) that the LLM assigns to specific tokens. It turns out that such an interface reveals much more information about the underlying model than previously thought, including exposing nonpublic architectural details of the LLM. Closed APIs, therefore, may provide a false sense of security to production LLM providers who may assume that their product information is private. At the same time, these capabilities empower the community with tools to audit LLM providers for bad behaviors such as unannounced model updates and abuse of open-source LLMs (Mökander et al., 2023) .In this paper we show that is possible to extract detailed information about LLM parameterization using only common API configurations. Our findings are centered on one key observation: most modern LLMs suffer from a softmax bottleneck (Yang et al., 2018) , which restricts the model outputs to a linear subspace of the full output space, as illustrated in Figure 1 . We call this restricted output space the LLM's image ( §2). This image can be obtained by collecting a small number of LLM outputs, and serves as a unique identifier or a signature for the model. We propose several algorithms that enable us to obtain the LLM image at low cost and speed for standard LLM APIs ( §3).Furthermore, we show that LLM images are useful in several applications, revealing important information about the model architecture. Concretely, we can exploit this image to provide an efficient algorithm for extracting full LLM outputs ( §4), to find the hidden dimension (embedding) size of the LLM ( §5), to detect and identify different kind of updates made to the model and to attribute model outputs to a specific model ( §6), among otherR d R v W ∆ v softmaxFigure 1 : LLM outputs are constrained to a low-dimensional subspace of the full output space. We can use this fact to glean information about API-protected LLMs by analyzing their outputs. Here we show how a toy LLM's low-dimensional embeddings in R d (illustrated here as a 1-D space) are transformed linearly into logits in R v (here, a 3D space) via the softmax matrix W. The resulting outputs lie within a (d = 1)-dimensional subspace of the output space. We call this low-dimensional subspace the image of the model. We can obtain a basis for the image of an API-protected LLM by collecting d of its outputs. The LLM's image can reveal non-public information, such as the LLM's embedding size, but it can also be used for accountability, such as verifying which LLM an API is serving. potential applications ( §7). We demonstrate several of our proposed applications in order to empirically verify their effectiveness. Notably, we design and implement an algorithm for finding the embedding size of an API-based LLM, and use it to estimate the embedding size of gpt-3.5-turbo (a closed-source API-protected LLM) to be 4096. Overall, our proposed methods have benefits for both LLM providers and for their clients. LLM providers may use their model images to establish unique identities for their models, thereby protecting their product and building trust with clients. Our proposed methods will also allow LLM clients to hold providers accountable for malicious behavior (Anderljung et al., 2023) . As a concrete use case in accountability, we demonstrate, using checkpoints from open-source LLMs that our LLM images can be used to attribute LLM output probabilities (or logits) to their generating model with high accuracy. The sensitivity of our LLM image to slight changes in the LLM parameters also makes them suitable for inferring granular information about the specific type of model update.Considering several proposals to guard access to LLM images, we find no obvious fix without dramatically altering the LLM architecture or making the API considerably less useful ( §8). Providers who choose to alter their API to prevent LLM image access risk removing interfaces with valuable and safe use cases for LLM clients. Though our findings could be viewed as a bug that LLM providers might feel compelled to patch, we prefer to view them as features that LLM providers may choose to keep in order to better maintain trust with their customers by allowing outside observers to audit their model. Ultimately, our results serve as a recommendation to LLM providers to carefully consider the consequences of their LLM architectures and APIs.• A method for extracting information about API-protected models, including the model's output space and embedding size. • Methods for extracting full-vocabulary logprob outputs from LLM APIs.• An estimate of the embedding size of an API-protected LLM (gpt-3.5-turbo).• An accelerated logprob extraction algorithm based on the LLM image.• An exploration of several other applications of our method for model accountability.Concurrently with our work, Carlini et al. (2024) propose a very similar approach for exposing details of production LLMs, though with a focus on defenses and mitigations against such attacks. The "final layer" that they extract in their attack corresponds to what we refer to in our paper as the model image. We view our papers as complementary, since our work emphasises practical applications of LLM images for better LLM accountability.MatMulW ∈ R v×d Softmax matrix Embedding h ∈ R d SoftMax Logits ℓ ∈ R v 0.6 0.2 0.1 0.1 0.0 Probabilities p ∈ ∆ v boyfriend queen thought breeze acrobatFigure 2 : A typical language model architecture. After the input its processed by a neural network, usually a transformer (Vaswani et al., 2017) , into a low-dimensional embedding h, it is multiplied by the softmax matrix W, projecting it linearly from R d onto R v to obtain the logit vector ℓ. The softmax function is then applied to the logit vector to obtain a valid probability distribution p over next-token candidates. 
llm outputs are restricted to a low-dimensional linear space: 
 The outputs of typical LLMs are naturally constrained to lie on a d-dimensional subspace of the full output space (Yang et al., 2018; Finlayson et al., 2024) . To understand this, begin by considering the architecture of a typical LLM (Figure 2 ). In this architecture, a transformer1 with embedding size d outputs a low-dimensional contextualized embedding h ∈ R d (or simply embedding). Projecting the embedding onto R v via the linear map defined by the LLM's softmax matrix W, we obtain logits ℓ = W h.Theorem 1 (Low-rank logits). LLM logits lie on a d-dimensional subspace of R v .Proof. Because W is in R v×d , its rank (i.e., number of linearly independent columns) is at most d. The rank of a matrix corresponds to the dimension of the image of the linear map it defines, i.e., the vector space comprising the set of possible outputs of the function. In other words, if the linear map w is defined asw(h) = W h, then w's image im(w) = {w(h) ∈ R v : h ∈ R d } is a d-dimensional subspace of R v .Thus, the LLM's logits will always lie on the d-dimensional2 subspace of R v .The low-dimensional restriction of logits actually translates into a similar restriction on LLM probabilities, which is useful since most LLM APIs return (log-)probabilities instead of logits. To understand why this is, we turn our attention to the model's next-token distribution p = softmax(ℓ). The softmax function3 transforms logits into a valid probability distribution, i.e., a v-tuple of real numbers between 0 and 1 whose sum is 1. The set of valid probability distributions over v items is commonly referred to as the v-simplex, or ∆ v .Theorem 2 (Low-rank probabilities). LLM probabilities lie on a d-dimensional subspace of ∆ v .Proof. Perhaps surprisingly, ∆ v is also a valid (v -1)-dimensional vector space (albeit under non-standard definitions of addition and scalar multiplication) and the softmax function is a linear map R v → ∆ v (Aitchison, 1982; Leinster, 2016) . Therefore, since dim(im(w)) = d, we also know that im(softmax •w) is a d-dimensional subspace of ∆ v .Thus, LLM output probabilities must also reside in a d-dimensional subspace.Finally, since ∆ v is an unconventional vector space, it is useful to translate LLM probability distributions into a more conventional vector space. To do so, we use the fact thatR v softmax(R v ) log softmax(R v ) clr (softmax(R v ))Figure 3 : Points in the logit space R v (far left) are mapped via the softmax function to points (probability distributions) on the simplex ∆ v (middle left). Crucially, the softmax maps all points that lie on the same diagonal (shown as points of the same color) to the same probability distribution. For numerical stability, these values are often stored as log-probabilities (middle right). The clr transform returns probability distributions to points to a subspace U v of the logit space (far right). The softmax function and clr transform are inverses of one another, and form an isomorphism between U v and ∆ v .∆ v is isomorphic to a vector subspace of R v via the softmax function. In particular, Fig-ure 3 illustrates how ∆ v is isomorphic to the hyperplane U v that is perpendicular to the all-ones vector 1 v . The isomorphism's inverse mapping ∆ v → U v is the center log ratio trans- form clr(p) = log p -1 v ∑ v i=1 log p i . By linearity, im(clr • softmax •w) is a d-dimensional subspace of U v ⊂ R v .Thus, LLM outputs occupy d-dimensional subspaces of logit space R v , probability space ∆ v , and U v . We call these subspaces the image of the LLM on each given space. A natural consequence this low-dimensionality is that any collection of d linearly independent LLM outputs form a basis for the image of the model, i.e., all LLM outputs can be expressed as a linear combination of these outputs. 
obtaining full outputs from api-protected llms: 
 As explained in the previous section, in order to obtain the image of an LLM, we need to collect outputs (token probabilites for each token in the vocabulary) from the LLM. Unfortunately, most LLM APIs do not return full outputs, likely because full outputs are large and expensive to send over an API, but perhaps also to prevent API abuse, since full LLM outputs contain lots of useful information (Dosovitskiy and Brox, 2016; Morris et al., 2023) and can be used to distill models (e.g., Hinton et al., 2015; Hsieh et al., 2023) . In their paper, Morris et al. (2023) give an algorithm for recovering full outputs from restricted APIs by taking advantage of a common API option that allows users to add a bias term β ≤ β max to the logits for specific tokens. The algorithm they describe requires v log(β/ϵ) calls to the API to obtain one full output with precision ϵ, or exactly v calls when the API returns log-probabilities of the top-2 tokens. Carlini et al. (2024) also propose a logprob-free method based on the same principle.We give a variant of their algorithm for APIs that return the log-probability of the top-k tokens that obtains full outputs in v/k API calls. We find that this improved algorithm suffers from numerical instability, and give a numerically stable algorithm that obtains full outputs in v/(k -1) API calls. We also give a practical algorithm for dealing with stochastic APIs that randomly choose outputs from a set of n possible outputs, a behavior we observe in OpenAI's API. This algorithm allows the collection of full outputs in nv/(k -2) API calls on average. Table 1 gives an overview of our algorithms with back-of-the envelope cost estimates for a specific LLM.Our goal is to recover a full-vocabulary next-token distribution p ∈ ∆ v from an APIprotected LLM. We will assume that the API accepts a prompt on which to condition the distribution, as well as a list of up to k tokens and a bias term β ≤ β max to add to the logits of the listed tokens before applying the softmax function. The API returns a record with the k most likely tokens and their probabilities from the biased distribution. For instance, querying the API with k maximally biased tokens, which (without loss of generality) we will identify as tokens 1, 2, . . . , k, yields the top-k most probable tokens from the biased distribution p ′ = softmax(ℓ ′ ) whereℓ ′ i = ℓ i + β max i ∈ {1, 2, . . . , k} ℓ i otherwise (1)and ℓ ∈ R v is the LLM's logit output for the given prompt.Assuming that the logit difference between any two tokens is never greater than β max , these top-k biased probabilities will be p ′ 1 , p ′ 2 , . . . , p ′ k . For each of these biased probabilities p ′ i , we can solve for the unbiased probability asp i = p ′ i exp β max -exp β max ∑ k j=1 p ′ j + ∑ k j=1 p ′ j (2)(proof in the §A.1). Thus, for each API call, we can obtain the unbiased probability of k tokens, and obtain the full distribution in v/k API calls.In practice, the algorithm described in §3.1 suffers from severe numerical instability, which can be attributed to the fast-growing exponential term exp β max , and the term ∑ k j=1 p ′ j which quickly approaches 1. We can eliminate the instability by sacrificing some speed and using a different strategy to solve for the unbiased probabilities. Without loss of generality, let p v be the maximum unbiased token probability. This can be obtained by querying the API once with no bias. If we then query the API and apply maximum bias only to tokens 1, 2, . . . , 1 -k, then the API will yield p ′ 1 , p ′ 2 , . . . , p ′ k-1 and p ′ v . We can then solve for the unbiased probabilities of the k -1 tokensEQUATION(proof in §A.2). By finding k -1 unbiased token probabilities with every API call, we obtain the full output in v/(k -1) calls total. Algorithm 1 gives a formal description of this method.Each of the above algorithms assume that the API is deterministic, i.e., the same query will always return the same output. However, this may not always be the case; for instance, we find that OpenAI's LLM APIs are stochastic. While this would seem to doom any attempt at obtaining full outputs from the LLM, it is possible to counter certain types of stochasticity.In particular, we model stochastic API behavior as a collection of n outputs p 1 , p 2 , . . . , p n Algorithm 1 Our numerically stable probability extraction algorithm, which takes a set of contexts, a set of tokens, and an API for an LLM with vocabulary V and returns the LLM's probabilities for each token in each context. The API takes a context and a set of token-bias pairs and returns the top-k biased probabilities. P (S) denotes the power set of a set S.function EXTRACT( contexts ⊆ V * , tokens ⊆ V, API : V * × P (V × R) → P (V × R) ) probs ∈ R |contexts|×|tokens| ▷ Initialize empty index to collect probabilities for c ∈ contexts do (v, p v ) = argmax (i,p)∈API(c,∅) p ▷ Get top probability token for each batch T ⊆ tokens with |T| = k -1 do ▷ Partition tokens into batches bias = {(i, β)} i∈T ▷ Set biases to β for batch tokens {(v, p ′ v )} ∪ {(i, p ′ i )} i∈T = API(c, bias) ▷ Get biased probabilities for i ∈ T do probs c,i = exp(log p ′ i -β -log p ′ v + log p v )▷ Get unbiased probability end for end for end for return probs end function from which the API randomly returns from. This might be the result of multiple instances of the LLM being run on different hardware which results in slightly different outputs. Whichever instance the API returns from determines which of the n outputs we get. In order to determine which of the outputs the API returned from, let p i v-1 be the second highest token probability for output p i , and observe that log p i v -log p i v-1 = log p i′ v -log p i′ v-1 for all outputs p i and biased outputs p i′ where tokens v and v -1 are not biased (proof in §A.3). Therefore, by biasing only k -2 tokens for each call, the API will return p i′ v and p i′ v-1 , which we can use to find log p i v -log p i v-1 , which serves as a unique identifier for the distribution. Thus, after an average of nv/(k -2) calls to the API we can collect the full set of probabilities for at least one of the outputs. 
fast, full outputs using the llm image: 
 The dominating factor in runtime for the algorithms in §3 is the vocabulary size v, which can be quite large (e.g., over 100 000 for OpenAI LLMs). We now introduce a preprocessing step that takes advantage of the low-dimensional LLM output space to obtain O(d) versions of all the above algorithms. Since d ≪ v for many modern language models, this modification can result in multiple orders of magnitude speedups, depending on the LLM. For instance, the speedup for a model like pythia-70m (Biderman et al., 2023 ) would be 100×. The key to this algorithm is the observation from §2 that d linearly independent outputs from the API constitute a basis for the whole output space (since the LLM's image has dimension d).We can therefore collect these outputsP = p 1 p 2 • • • p d ∈ ∆ dv as a preprocessing step in O(vd) API calls using any of the above algorithms and d unique prompts, and then use these to reconstruct the full LLM output after only O(d) queries for each subsequent output.To get a new full output p, use any of the above algorithms to obtain p 1 , p 2 , . . . , p d . Since p resides in a d-dimensional space spanned by the columns of P, the rest of the values of p are fully determined by these first d values. §B gives the details of how to solve for the remaining values. Thus we can retrieve p in only O(d) API queries. This (v/d)× speedup makes any method that relies on full model outputs significantly cheaper so long as the number of model outputs needed exceeds d. Such methods include model stealing (Tramèr et al., 2016; Krishna et al., 2019) which attempts to learn a model that exactly replicates the behavior of a target model, and LM inversion (Morris et al., 2023) which uses LLM outputs to reconstruct hidden prompts. Additionally, the preprocessing Algorithm 2 Our algorithm for finding the model image and embedding size for an APIprotected LLM. To get the embedding size of the model without finding the image, extract only the first t token probabilities at each step. With API caching this takes O(vd/k) time, or O(d 2 /k) to only extract the embedding size. step can be computed once then shared between any number of clients, further diluting the cost of full outputs.: V * × P (V × R) → P (V × R) ) for t = 1, 2, . . . do L t = clr (EXTRACT([t], V, API)) ∈ R t×|V | ▷ Get 
discovering the embedding size of api-protected llms: 
 Assuming only the generic output layer described in Figure 2 , it is possible to infer the embedding size d of an API-protected LLM from its outputs alone. Since the model outputs occupy a d-dimensional subspace of ∆ v , collecting d linearly independent outputs p 1 , p 2 , . . . , p d from the LLM forms a basis for the LLM's image. In other words, subsequent model outputs will be a linear combination of the first d outputs. We can therefore discover the value of d by collecting outputs one at a time until the number of linearly independent outputs in the collection (i.e., the rank) stops increasing, which will occur after we have collected d prompts. We find that it suffices to slightly over-collect, e.g., d + 1000 outputs. Algorithm 2 formalizes this procedure.To find the rank of model outputs, we use the fact that a matrix with d linearly independent columns will have d non-zero singular values. We use singular value decomposition to obtain the singular values of the matrix L = clr(p 1 ) clr(p 2 ) • • • clr(p d+1000 ) and observe the index at which the magnitude of the singular values drops to zero, which occurs at index d. In practice, numerical imprecision causes the magnitudes drop almost to zero.To validate our method, we collect next-token distributions (conditioned on unique, 1-token prompts) from several open-source LLMs from the Pythia family (Biderman et al., 2023) with embedding sizes ranging from 512 to 1024. For all these models, the singular values of the resulting output matrix drop precisely at index d, as shown in Figure 4 . To demonstrate our method's effectiveness against API-protected LLMs, we use our algorithm from §3.3 to collect nearly 6,000 next-token distribution outputs from gpt-3.5-turbo4 , a popular API-protected LLM whose embedding size is not publicly disclosed. We find that this model's singular values drop between index 4,600 and 4,650, indicating that the embedding size of this model is at most this size. This predicted embedding size is somewhat abnormal, since LLM embedding sizes are traditionally set to powers of two. If this were the case for gpt-3.5-turbo, it would be reasonable to guess that the embedding size is 2 12 = 4096. We predict that our raw estimate of 4600-4650 is an overestimate of the true embedding size, since any abnormal outputs due to errors (whether in our own code or OpenAI's) would only increase the dimensionality of the observed output space. For instance, if we inadvertently collected 504 corrupted outputs, then a model with embedding size 4096 would appear to have an embedding size of 4600.Knowing the embedding size alone is insufficient to ascertain an LLM's parameter count. However, since most known transformer-based LLMs with embedding size 4096 have around 7 billion parameters, it is likely that gpt-3.5-turbo has a similar number of active parameters. Any other parameter count would result in either abnormally narrow or wide models, which often perform worse. The actual parameter count may be much higher if the model uses a "mixture-of-experts" architecture (Shazeer et al., 2017) . Given the active development and decreasing cost of inference with gpt-3.5-turbo, it is possible that its size and architecture has changed over time. Fortunately, our method can be used to monitor these updates over time, alerting end-users when LLM providers change embedding size (and presumably therefore, model size). 
attributing model outputs and auditing model updates: 
 An LLM image can serve as an identifying signature for the model, as shown in Figure 5 , where the logit output from a pythia-70m checkpoint lies uniquely in the checkpoint's image, and not in the image of the preceding or following checkpoints, nor the checkpoints of any other similar model.5 This suggests that LLM images are highly sensitive to parameter changes, which makes sense because the intersection of two different d-dimensional spaces is an even lower-dimensional space. Thus, it is possible to determine precisely which LLM produced a particular full-vocabulary output using only API access to a set of LLMs, and notably, without knowing the exact inputs to the model.This finding has implications for LLM auditing (Mökander et al., 2023) , such as obtaining granular information about updates to API-protected LLMs. For instance, if the LLM image remains the same but the logit outputs change, this would indicate a partial model update where some part of the model changes but the softmax matrix remains the same. This might happen when the LLM has a hidden prefix added to all prompts and this hidden prefix changes, or when some part of the model was updated while leaving the softmax Table 2 : Implications of image/logit changes. matrix unchanged. Table 2 gives an overview for how to interpret various combinations of detectable API changes.Using the same principle, one can also detect malicious use of open-source LLMs. If a provider attempts to profit off of an open-source model with a non-commercial license, an auditor can reveal the scheme by checking the LLM image, even if the provider used a hidden prompt to mask the model identity. It would be unlikely for images of two different LLMs to match, unless purposely designed to do so. Note that this test is one-sided, and malicious providers may slightly fine-tune an LLM to avoid detection. 
more applications: 
 Access to the LLM's image unlocks several additional capabilites, some of which we review below. We leave further investigation of these methods for future work.Access to the LLM image can afford even finer granularity insight into LLM updates. For instance, as LoRA (Hu et al., 2022) is a popular parameter-efficient fine-tuning method which adjusts model weights with a low-rank update AB where A ∈ R v×r and B ∈ R r×d so that the softmax matrix W ∈ R v×d becomes W + AB. It may be possible to detect these types of updates by collecting LLM outputs before (L ∈ R v×d ) and after (L ′ ∈ R v×d ) the update and decomposing them as W H = L and (W + AB)H ′ = L ′ where H, H ′ ∈ R d×d . If such a decomposition is found then it is likely that the weights received a low-rank update.We leave it to future work to find an efficient algorithm for this decomposition.Due to the low-rank constraints on LLM outputs, it is possible that some tokens become unargmaxable (Demeter et al., 2020; Grivas et al., 2022) , i.e., the token always has less probability than some other token. This happens when the LLM's embedding representation of the token lies within the convex hull of the other tokens' embeddings. Previously, finding unargmaxable tokens appeared to require full access to the softmax matrix W, however, it is possible to identify unargmaxable tokens using only the LLM's image, which our method is able to recover. This technique allows API clients to find tokens that the model is unable to output and thereby elicit unexpected model behavior.One might use the image to approximately reconstruct the output layer parameters. We hypothesize that LLM embeddings generally lie near the surface of a hypersphere in R d with a small radius r. We see evidence of this in the fact that the Pythia LLM embedding norms are all small and roughly normally distributed, as shown in Figure 6 in the Appendix. We can attempt to recover W up to a rotation by assuming that all embeddings must have unit magnitude, then, given a matrix L ∈ R n×v of model logits, we can find W (up to a rotation) by finding a decomposition W H = L such that for all i, ∥W i ∥ 2 = 1. This solution may also be approximated by finding the singular value decomposition W ΣV ⊤ of L, though rows of this W will have magnitude less than 1.In a recent paper, Finlayson et al. (2024) propose a decoding algorithm that avoids type-I sampling errors by identifying tokens that must have non-zero true probability. Importantly, this method relies on knowing the basis of the LLM's output space, and is therefore only available for LLMs whose image is known. Our approach for finding the model image makes this decoding algorithm possible for API LLMs. 
mitigations: 
 We consider three proposals that LLM providers may take to guard against our methods. The first proposal is to limit or entirely remove API access to logprobs. This is theoretically insufficient, as Morris et al. (2023) show that it is possible to obtain full outputs using only information about the biased argmax token, albeit inefficiently in v log(β/ϵ) API calls.Providers may rely on the extreme inefficiency of the algorithm to protect the LLM, as OpenAI appeared to do in response to Carlini et al. (2024) . However, our algorithm in §4 brings the cost down to a feasible O(d log(β/ϵ)) API calls per output once the initial work of finding the LLM image finishes, the result of which can be made public and reused.The second proposal is to remove API access to logit bias. This would be effective, since there are no known methods to recover full outputs from such an API. However, logit bias has several useful applications, such as for blocking undesirable tokens and controlling text generation, and making such a restriction could seriously degrade the usefulness of the API.Lastly, we consider alternative LLM architectures that do not suffer from a softmax bottleneck. There are several such proposed architectures with good performance (e.g., Xue et al., 2021; Yu et al., 2023; Wang et al., 2024) . Though this is the most expensive of the proposed defenses, due to the requirement of training a new LLM, it would have the beneficial side effect of also treating other tokenization issues that plague large-vocabulary LLMs (e.g., Itzhak and Levy, 2022) . A transition to softmax-bottleneck-free LLMs would fully prevent our attack, since the model's image would be the full output space. 
conclusion: 
 We have shown how the low-rank constraints imposed by the softmax bottleneck expose non-public information about API-protected LLMs. We also demonstrated how this information can be used to efficiently extract full model outputs, expose model hyperparameters, function as a unique model signature for auditing purposes, and even detect specific types of model updates, among other uses. We find that some current protections against these attacks are insufficient, and that the more effective guards tend to inhibit sanctioned API use cases (i.e., logit bias), or are expensive to implement (i.e., changing model architecture).Overall, we find that the benefits of our proposed methods outweigh the harms to LLM providers. For instance, allowing LLM API users to detect model changes builds trust between LLM providers and their customers, and leads to greater accountability and transparency for the providers. Our method can also be used to implement efficient protocols for model auditing without exposing the model weights. On the other hand, discovering the embedding size of the LLM does not enable a competitor to fully recover the parameters of the LLM's softmax matrix or boost performance of their own model. Even efficiently extracting full outputs for model stealing or inversion ( §4) is unlikely to have detrimental effects, as these are already known threats and are only made more efficient by our methods. We therefore believe that our proposed methods and findings do not necessitate a change in LLM API best practices, but rather expand the tools available to API customers, while informing LLM providers of what information their APIs expose. 
 
 Appendix: This appendix contains derivations for the equations used to solve for unbiased token probabilities given biased LLM outputs.Our goal is to provep i = p ′ i exp β max -exp β max ∑ k j=1 p ′ j + ∑ k j=1 p ′ j (4)Proof. We begin with the definition of the softmax functionEQUATIONWe then rearrange to obtainEQUATIONthe left hand side of which is independent of the bias term, meaning it is equivalent to when b max = 0, i.e.,EQUATIONWe EQUATIONwhich concludes the proof. form a basis for a d-dimensional vector subspace of R v-1 , and alr(p) lies within this subspace. Therefore, there is some x ∈ R d such that alr(P)x = alr(p). To solve for x, all that is required is to find the unique solution to the first d rows of this system of linear equations (28)After finding x, we can reconstruct the full LLM output p = alr -1 (alr(P)x), where the inverse alr function is defined as alr -1 (x) = 1 1 + ∑ v-1 i=1 exp x i • (1, exp x 1 , exp x 2 , . . . , exp x v-1 ) .(29)Technically, any neural network suffices here.More accurately, the logits will always lie on an at-most-d-dimensional subspace. For convenience, we assume full-rank matrices, and thus a d-dimensional subspace.Our use of "softmax" here corresponds to temperature softmax with temperature 1.We specifically use model version 0125, accessed February 1-19, 2024.Open source models with detailed checkpoint information (e.g.,Biderman et al., 2023;Liu et al., 2023) make this type of detailed analysis possible. 


Decision: The paper studies how the logits of a black-box LLM (such as OpenAI’s models) can reveal non-public information about the model. The key idea is that most modern LLMs’ outputs are limited to a low-dimensional subspace of the full output space. Combined with this paper’s another contribution on an improved algorithm to recover the full probability distribution with limited API output, it provides a very efficient way to recover the softmax layer and the rank of it, which consequently leads to various non-public information about the model such as the hidden dimension of the model (which could hint the model size) and the nature of the model updates (e.g., whether or not the softmax layer was updated).

Reviewers acknowledged that the paper is technically solid and made a clear algorithmic contribution under various assumptions such as the availability of logprobs, stochastic changes, etc  (Reviewer CQvU, gbAq). Reviewers also think the paper opens up a broader discussion on forcing more transparency from API providers on silent backend changes, employing this method for model auditing, and developing mitigation for this type of attacks (Reviewer CQvU, gbAq). Reviewers also think the findings are interesting and the revealed information about OpenAI models is valuable (Reviewer CQvU, gbAq, D4ia).

Some concerns have been raised regarding the practical implications of the findings, given that the attacks are defensible, the information it leaks are limited (Reviewer gbAq, D4ia), and it relies on specific assumptions like the presence of logit bias features (Reviewer gbAq).

[At least one review was discounted during the decision process due to quality]

Reviews: review: paper_summary: paper_summary: This paper explores the vulnerabilities in the API deployment of large language models, specifically focusing on how a limited number of API queries can expose non-public information about such models. The authors demonstrate that softmax bottleneck deduces extensive details about a model's structure, such as its embedding size. The study further suggests methods for LLM providers to shield against these vulnerabilities, while also proposing that these weaknesses could enhance transparency and accountability in LLM operations.
strength_weakness: strength_weakness: + The attack can be easily defended by blocking logits or adding noise to the outputs. 

+ Although several types of attack has be discussed in this work, but the main outcomes and purpose of this attack is unclear. 
How these findings can be orchestrated as a comprehensive attack that harms LLM applications. I would consider discovering size of h in LLM is minor as it is just a number and we still need to train the models. Recovering the exact logits of the top K tokens is still minor given the 'biased' top K logits already provides essential information.

+ Improvements are needed in the clarity and organization of the writing:
  + A comprehensive overview of the algorithm and underlying theory would enhance understanding.
  + Clarification is needed on how this work differs from [1]. Specifically, if Sections 7 and 8 are intended to address these differences, a more thorough analysis in these sections could help establish a clear distinction from prior work

+ The suggested applications (in Section 7) lack sufficient detail and are not adequately supported by theoretical frameworks or experimental validation.

+ The countermeasures suggested are not substantiated by experimental evidence.

+ The paper was submitted several weeks after [1], (and arXiv version is also several days after [1]) yet it does not adequately distinguish its theoretical contributions or empirical analysis from those found in [1]. For example, an algorithm to recover the last layer was proposed in [1] but not discussed in this work. I would consider recovering the 'shape' is far less interesting than recovering 'parameters' of the linear module. This means the missing piece to [1] is the most essential part of this work. 

I am aware that this is a concurrent work to [1], and my evaluation is based on the quality of this paper.
paper_summary: paper_summary: This paper discussed an interesting method to partially recover some important meta data of black box LLM, primarily the hidden dimension before embedding matrix. 

The core idea is that the output logit vectors reside in dimension $d$ which is much smaller than vocabulary size $V$, thus a singular value decomposition over a collection of inferred logits matrix successfully recovers the actual hidden dimension.

The authors further discussed extensions such as inferring the model that generates certain outputs or auditing for any model updates, etc.
strength_weakness: strength_weakness: The primary concern on this paper is the contribution on top of [Carlini 2024]. After comparing with two papers I feel like authors need to make it clearer on what additional value this paper brings to research community. Right now the authors did not have such comparison thoroughly. Thus, I would like the authors to include a "Contribution" section to clarify this.

Is the main breakthrough of this paper lies in the algorithmic complexity of doing LLM invocations? If so, I expect to make it clear by comparing big Os of this paper vs. [Carlini 24].

Though I appreciate the detailed discussions on potential future applications, I doubt the feasibility of quite a few. For instance "detecting the LoRA updates", the reality is people could simply skip lora updates on embedding layer; and the wording "It may be possible to detect these types of updates" sounds sneaky. It should be avoided in academia papers.
questions: questions: My questions are mostly tied to "reasons to reject" section. Please respond to that section.
paper_summary: paper_summary: This paper presents an algorithm to extract proprietary LLM information from an API-protected black-box LLM.

The algorithm uses the observation that LLM outputs are "softmax-bottlenecked", or they occupy only a low-dimensional subspace of the `R^vocab_size` output space. Using this observation, along with logit bias access to APIs, the algorithm can efficiently extract this low-dimensional subspace. This subspace can be leveraged to 1) infer the hidden dimension of the softmax layer; 2) extract a fingerprint of the API, which can be used to monitor API updates (like LLM changes).

The algorithm is tested in a real-world setting on the OpenAI GPT3.5 Turbo model, and is shown to work with high accuracy.
strength_weakness: strength_weakness: This is a great paper overall and a good fit for COLM. I have a few concerns, but overall I think the strengths outweigh the weaknesses.

1. The method (as well as prior works), rely on API providers allowing users to adjust logit biases. Based on Section 8, this method will not work without logit biases. However, I think logit bias is a fairly limited hack from OpenAI's perspective, rather than a long-term solution for controlled decoding. With rapidly improving instruction following capabilities, I think models maybe able to achieve effects similar to logit bias with just natural language instructions in the prompt, and get more natural-looking output. For reference, I don't see logit bias access to the Claude API (https://docs.anthropic.com/claude/reference/messages_post) or the Gemini API (https://ai.google.dev/api/python/google/generativeai/GenerationConfig).

2. Overall, I think the amount of information leakage from the attack is fairly limited, which limits the implications of the work. The only major information extracted is the dimensionality of the softmax matmul layer, which leaves an auditor speculating about the rest of the giant LLM (the more important proprietary information). The signature / fingerprint aspect of the attack is definitely interesting, but Section 6 and Table 2 makes me think there are too many variables that could change the API signature. This could include innocuous changes like timestamp in the hidden prompt, or search results in the hidden prompt of a RAG-enabled API. Watermarking, routed models, prompt preprocessing, output post-processors (safety filters?) could also change signatures (see questions below). Given that the APIs are rapidly changing due to industrial competition, frequent changes in API signature could leave auditors confused.

3. (minor, likely an unreasonable request) I was a bit surprised that experiments were only done on one proprietary model, and one family of open-source models. In particular, were there some cost bottlenecks to run the experiment on GPT-4 Turbo models?
questions: questions: Questions:

- What happens if the logits are postprocessed before being sent to the softmax? In particular, I think the addition of a watermark [1] could change the image of the LLM without changing the underlying LLM model. I also have a similar question for temperature changes in the API.

- How well does the method work with APIs which are not a single LLM (they route queries between two or more LLMs, like [2, 3, 4]). Does that muddy the image procured by this method, possibly adding to the estimated "effective dimension"? What if routing weights are changed? Similar question for the mixture-of-softmax method from [5].

Typos / presentation:

- The paper is missing a related work section, and I think more references would help in better grounding the work. Notably, the work is closely related to hyperparameter extraction and model extraction efforts (some examples: [6, 7, 8, 9]).

- page 3: "we turn out attention" --> "we turn our attention"

- page 3/ 4: Figure 3 is confusing and I don't think the translation to the clr space is helping my understanding (possibly better for the Appendix). Please correct me if I'm wrong, but my understanding reading the text is that everything after the softmax matmul matrix (logits, softmax, log-softmax) are in a d-dimensional subspace of R^v. Figure 3 feel like there is further reduction in image dimensionality from logits --> softmax-out (which I don't think is true).

Here's an alternative suggestion: Present a table with three columns being (transformer operation, vector space dimension, effective image dimension). Something like the table below:

(transformer operation, vector space dimension, image dimension)  
(end of transformer sub-blocks, R^d, R^d),  
(softmax MatMul `l = Wh`, R^V, R^d),  
(softmax, `p = e^l / sum(...)`, R^V, R^d),  
(log-softmax, `q = log(p)`, R^V, R^d)  
etc.

Accompany this table with a simulation / points in a space where V = 3, d = 2, and one figure accompanying each of the rows above. The first row could be a 2-d plot, and all others 3-D plots with the points in a 2-d surface.

[1] - https://arxiv.org/abs/2301.10226  
[2] - https://arxiv.org/pdf/2310.12963  
[3] - https://arxiv.org/abs/2207.07061  
[4] - https://arxiv.org/abs/2211.17192  
[5] - https://arxiv.org/pdf/1711.03953  
[6] - https://arxiv.org/abs/1711.01768  
[7] - https://arxiv.org/abs/1802.05351  
[8] - https://arxiv.org/abs/1910.12366  
[9] - https://arxiv.org/abs/2305.15717

Comments: title: Thanks for your response
Response
Thank you for your comments, keeping my score
comment: As the reviewing period approaches its end we thank you for your input and reviewing our paper. Did our responses (see our above comment) address your concerns? In particular, you mentioned that your primary concern is the concurrency with Carlini et al. (2024). In light of the COLM concurrency policy, does this change your evaluation score?
We have to admit that the concurrent work [1] is more preferred and mature, because it 1) involves more solid theories, 2) is motivated by a concrete attack goal and corresponding threat model, and 3) is published several days earlier than this submission. I consider this work concurrent to [1] and my decision is independent to 3.

I still do not feel the properties proposed in this work pose a solid threat to LLM, but they are interesting properties to NLP community (probably worth discussing in COLM). I would suggest improving this work to form a solid attack, but considering the timing, I am not against the work to be published to COLM for the awareness of such threats in NLP community.
Thank you for getting back to us so quickly and giving more details about your concerns. Here is some additional relevant information.

1. Yesterday COLM organizers clarified the related work policy in an email to reviewers and authors:
> Related work concurrency: we follow the NeurIPS policy with regards to concurrency and related work. Papers appearing less than two months before the submission deadline are generally considered concurrent to NeurIPS submissions. **Authors are not expected to compare to work that appeared only a month or two before the deadline.**

2. Both OpenAI and Cohere (not sure about others) were vulnerable to our method when we wrote the paper. OpenAI changed their API in response to becoming aware of the attack. Cohere's API was patched around the same time, presumably for the same reason. These responses indicate that these companies take our findings seriously. **In terms of longer term impact, API providers are now aware of the vulnerabilities going forward and will design their interfaces with this in mind.**

Do these additional details address your concerns?
Thanks for your response.

I checked the timestamps of both papers, seems indeed a concurrent case. I'll maintain my rating because 1/ COLM review guideline didn't mention how to deal with such senario. 2/ As today's APIs either do not provide logits (as Anthropic) or patched (as OpenAI), it's not very clear to me about the longer term impact. But other than that I think this paper is solid and interesting enough.
Thank you for your detailed replies! I will keep my original score, and continue to support the acceptance of the paper.