Better Language Model Inversion by Compactly Representing Next-Token Distributions
Authors: Murtaza Nazir
Matthew Finlayson
John Xavier Morris
Xiang Ren
Swabha Swayamdipta
Venue: NeurIPS 2025 poster ()
Keywords: Language model inversion
language model
adversarial attack
prompt injection
jailbreak
inversion
system prompt
softmax bottleneck
Primary area: deep_learning
TL;DR: This paper introduces PILS, a novel language model inversion method that leverages the low-dimensionality of next-token distributions, enabling their lossless compression over multiple generation steps for markedly improved prompt recovery.

Abstract: Language model inversion seeks to recover hidden prompts using only language model outputs. This capability has implications for security and accountability in language model deployments, such as leaking private information from an API-protected language model’s system message. We propose a new method – prompt inversion from logprob sequences (PILS) – that recovers hidden prompts by gleaning clues from the model’s next-token probabilities over the course of multiple generation steps. Our method is enabled by a key insight: The vector-valued outputs of a language model occupy a low-dimensional subspace. This enables us to losslessly compress the full next-token probability distribution over multiple generation steps using a linear map, allowing more output information to be used for inversion. Our approach yields massive gains over previous state-of-the-art methods for recovering hidden prompts, achieving 2–3.5 times higher exact recovery rates across test sets, in one case increasing the recovery rate from 17% to 60%. Our method also exhibits surprisingly good generalization behavior; for instance, an inverter trained on 16 generations steps gets 5–27% higher prompt recovery when we increase the number of steps to 32 at test time. Furthermore, we demonstrate strong performance of our method on the more challenging task of recovering hidden system messages. We also analyze the role of verbatim repetition in prompt recovery and propose a new method for cross-family model transfer for logit-based inverters. Our findings suggest that next-token probabilities are a considerably more vulnerable attack surface for inversion attacks than previously known.

Better Language Model Inversion by Compactly Representing Next-Token Distributions
Murtaza Nazir
Matthew Finlayson" University of Southern California
John X. Morris Cornell University
Xiang Ren
Swabha Swayamdipta
University of Southern California
University of Southern California
Abstract
Language model inversion seeks to recover hidden prompts using only language model outputs. This capability has implications for security and accountability in language model deployments, such as leaking private information from an APl-protected language model's system message. We propose new method- prompt inversion from logprob sequences (PILS) -tat recovers hidden prompts by gleaning clues from the model's next-token probabilities over the course of multiple generation steps. Our method is enabled by a key insight: The vector- vector-valued outputs of language model occupy a low-dimensional subspace. This enables to losslessly compress the full next-token probability distribution over multiple generation steps using linear map, allowing more output information to be used for inversion. Our approach yields massive gains over previous state-of-the-an the-art methods for recovering hidden prompts, achieving 2-3.5 times higher exact recovery rates across test sets, in one case increasing the recovery rate from 17% to 60%. Our method also exhibits surprisingly good generalization behavior; for instance, inverter trained on 16 generations steps gets 5-27 points higher prompt recovery when we increase the number of steps to 32 test time. Furthermore. we demonstrate strong performance of our method on the more challenging task of recovering hidden system messages. We also analyze the role of verbatim repetition prompt recovery and propose new method for cross foosslfamily model transfer for based inverters. Our findings show that next-token probabilities are considerably more vulnerable attack surface for inversion attacks than previously known.
Introduction
The task of language model inversion is to recover unknown prefix string (hidden prompt), given only information about a language 2odel's² outputs, conditioned on that prefix. This capability can potentially be used to steal hidden prompts, leak private information, or (on the flip side) detect malicious prompts that could cause harmful behavior in language models. Advancements inversion, thus have important implications for language model security and accountability. Prior work in language model inversion leverages information in next-token (log-) known logprobs- text outputs [34, 12], or employing prompt-based attacks [35]. However, these methods have shown only modest success. For example, state-of-the-ar methods recover fewer than one-in-four Llama Chat prompts from in-distribution evaluation sets, and fare much worse on out-of of of--iitribttton prompts.
Correspondence to themurtazanazire gmail.com and mflillys@@ usc.edu In this work, only concern ourselves with causal language models as inversion targets.
39th Conference on Neural Information Processing Systems (NeurIPS 2025).

---

Black box
Logprobs
Architecture
Inverter input
Hidden prompt
Compress
Recovered prompt
Language model
Inverter
Compression
RD Unembed
RV
softmma
R
RD
Hidden state h
Logits
Probabilities
air logits
Inverter imput
Black box
Linese
Figure 1: Our goal is to recover a hidden prompt based on the outputs of a black box language model. To do this, take a sequence of logprobs, losslessly compress them into a sequence of T low-dimensional vectors, and feed them into an encoder-decoder inverter model, which outputs the recovered prompt. Our compression method takes advantage of the fact that model outputs are linear projections of the language model's D-dimensional final hidden state (see $3.1).
This work aims improve the performance and generalizability of language model inversion, with focus on logprobs-band based inversion, since logprobs contain rich information about model outputs. Surprisingly, the best-known logprobs-basee bbased method, Logit2Text or L,T [21], lags behind more recent text-based inversion methods [34]. Notably, L2T only uses language model outputs from
single generation step, since logprobs are expensive to obtain from typical language model APIs and
require a lot of space- eaach logprob is vector of dimension equal to the vocabulary size of the target model, which can hundreds of thousands of tokens.
We propose method to overcome the high representation size and API costs of L2T. As illustrated in Figure 1, we apply lossless compression the target model's logprob outputs (at multiple generation steps) to obtain compact representations with dimension equal to the target model's 's embedding size D. We confirm empirically that these representations are good approximation of the full logprobs, by showing that an inverter that uses them performs as well as L2T (and slightly better). The key insight of our method is that logprobs live in D-dimensional subspace, meaning that we can compress them with a simple linear map. Furthermore, obtaining these compact representations requires only D logprob values from the target model, greatly reducing the API cost by 1-2 orders of magnitude.
With this improved representation scheme, we propose a new inversion method, prompl inversion from logprob sequences (PILS), that incorporates target model outputs from multiple generation steps as imput to our inverter. The intuition behind our approach being effective is that the target model may surface information about certain parts of the prompt until later in the generation. We find that our method massively improves performance on inversion, and boasts an exact recovery rate 2-3.5x higher than the previous -fate-of-the-art for both in-domain and out-of-domain prompts. We also find that our trained inverters exhibit surprisingly good generalization: an inverter trained on 16 generation steps continues improve as we increase the number of steps beyond 16 at test time, Finally, we leverage our compact representations to propose method to adapt our inverter to new models without any additional training (model transfer), novel transfer method for logprob-based inverters."
Related work
Broadly speaking, model inversion attempts recover neural network inputs based on their vector- valued outputs. Inverters for vision models [19, 7, 29] use image classifier logits. Inverting language embedding models is also possible, recovering text inputs from vector-valued sentence and document embeddings [26, 16, 20]. Morris et al. [21], introduced L,2T, the first (to our knowledge) method for recovering hidden prompts from language model logprobs; method builds this work, contributing a compact representation of language model outputs.
Language model inversion has received attention within the broader field of red-teaming [30], where adversaries attempt to elicit undesirable behaviors from language model in limited-access (e.g., API)
2 Our code is available at https: //tp::://ttthbb himm/Dill-aabll Lab/PIIS.

---

settings. Existing methods use pomptt-aseed jailbreak and injection attacks to coax the language model to output its hidden system message verbatim [35, 32]. Unlike our work, these methods generally rely on discrete text-valued model outputs and generally do not involve training an inversion model.
Our technical contributions constitute an application of the low-rank constraints that transformer language model outputs are subject to, known as the softmax bottleneck [33]. This fact has previously been used to discover unargmaxable tokens in language models [13], prevent sampling errors during text generation [9], and uncover hidden architectural details of API-protected language models [10, 5]. As a way of relaxing the requirement of logprob full access for inversion, Zhang et al. [34] and Gao et al. [12] combine aspects of both text- baxt-base system message discovery and language model inversion. Our method shares this goal but takes an intermediate approach where we drastically reduce the number of logprobs needed rather than eliminate them altogether. We use the Output2Prompt (O2P) [34] and Logit2Texx (I.2T) [21] as the main baselines for comparison with our method.
3 Preliminaries
We establish some notation, assumptions, and mathematical background for our method. We assume a typical language model architecture with embedding size D, and vocabulary size V. At every generation step, the model produces hidden state h RD, which is multiplied by the model's unembedding matrix W to obtain logits f = Wh E RV, which are normalized via the softmax function to obtain probabilities p = softmax((). The entries of p are interpreted probability for each token in its vocabulary. Generation typically proceeds by sampling according to the probabilities in p. by greedily picking the most-probable token each generation step.
the model's predicted
3.1 Language model outputs are losslessly compressible
We now show how it is possible to recover the hidden state of a language model from its probability output p up to linear transformation. This demonstrates exactly how we compress the logprobs of the language model in our proposed method ($4).
Theorem 1. If language model with hidden size D, vocabulary size V, and unembedding matrix W, generates a hidden state h and outputs p Softmax(Wh) then for any set of indices D S
{1.2..... V) we have that air( p) E R2 is a linear transformation of h.
Proof. Probability vectors p have the property that all entries are in the range (0. 1) and that the entries sum to 1. It is a lesser known fact that the set of valid probability distributions over V items-known the simplex, A formos a vector space, albeit with non-standard definitions of addition +A and scalar multiplication [15]. In particular, for vectors p and q in A", addition is defined as p +A q = (p1q).... pvqvv/E EYE piqi: and for a scalar d E R, multiplication is defined as l ' p (p²- (p/......P)//E"" //E/ D.- Under this definition, one can check that the softmax function satisfies linearity [8], which means it is linear map RV - AV. Additionally, the simplex A is isomorphic to R R via the additive log rutio transform air( = log P1:(v-1) - log pv, as shown in Aitchison [1].a In other words, air is also a linear function and maps the probabilities of the simplex back into a standard vector space.
We will now show that it is possible to recover the hidden state h from the logprob outputs of a model (up to linear transformation), as shown in Figure 1. Letting w be the linear map x Wx, we have that the w : R² + RV, softmax : RV - AV. and air A - R are linear. It must therefore be the case that air o softmax ow RD RV-1 is lincar and can be parameterized by matrix A E (V-1)), The implication here is that applying the air transform to a language model output and then applying a full-rank linear down-projectio of our choice (say, by dropping all but D indices) we can recover the final hidden state of the model, up to an multiplication of a D x D matrix. This is because for any set D of D indices, alr(softmax (Wh) Wh)),, > = A h.
While it is possible that Ag has less than full rank, in which case the recovered hidden state loses information, we easily avoid this in practice ($4). Thus, a language model outputs probabilities p. we know that air( p) l can linearly encode all the information in the final hidden state h.
We use NumPy-like indexing notation, where xab (Na. Naths ... xb) and x (1,1,8) xx))
3

---

Logprobs
Logprobs
Linear
model [3] with embedding Inverter input (1.2T [21])
Inverter input (PILS; Ours)
Figure 2:
comparison between L2T [left: 21] and method PILS (right). A language model produces a sequence of logprob vectors in R L2T takes only the first vector and reshapes it to fixed sequence length of [V/D]. padding with 0 needed. PILS losslessly compresses logprobs into R", and uses multiple generation steps as input to the inversion model.
3.2 Threat model
We consider the scenario where an attacker has limited access to a language model with embedding size D (as through its model API). In particular, the attacker can obtain the logprobs log of fixed set of D) tokens for each generation step of the language model. The attacker can observe language model outputs conditioned any prompt of their choosing, or conditioned hidden prompt. The goal of the attacker is to discover the hidden prompt.
As one example, this threat model is consistent with the OpenAI language model APIS which offers logit bias, greedy decoding, and the logprob of the most-likely token. this setting, it is possible to obtain the logprob for target token by first noting the logprob log p of the most likely token, performing a bisection search to find the minimum logit bias 8 that causes the model to select the target token, then calculating the logprob for the target token as B log [10, 21]. This method allows users to find the logprob of the target token with precision in Oloog 4 API queries.
4 Language model inversion from compressed logprobs
The main contribution of our method is finding a way to compress and feed a 7 X V language model output to the inversion model. Previous work [21] approached this problem by using only a single generation step (T' = 1) and reshaping the resulting V-length vector into a sequence of -enveh-eength vectors (Figure 2: left). Our method independently compresses each V-length generation vector into D-length vector, then passes T such vectors to the inverter (Figure 2; right).
Compressing logprobs Our target model outputs a sequence of logprobs log p ("),...,, log p (T) RV. Following our insights from $3.1, we can recover the hidden states of the model (up to multiplication by an unknown D X matrix) by taking the air transform of the probabilities and dropping all but entries to get (1) hhhh (T) E RR, where (0) air( p (7) I.D.. practice, we find inverter performs better when using random set of /) 100 tokens rather than the first D) likely due to some of the first D) tokens having (almost) linearly dependent embeddings, which causes the compression to become degenerate.
Inverter architecture As our learned inverter, we use an
size Dinvert- The encoder takes the sequence recovered hidden states 4 (1) (T) E RD as input embeddings, and the decoder generates the hidden prompt. To address potential mismatches between the embedding size of the target model Dand inverter model Dinvert we add a learned feed-forward adapter layer with hidden size D. dropout [27], and GELU nonlinearity [14] before the encoder input layer. We use single-layer feed-forward network because a less expressive linear function would lead to information loss when D D inverrt-
Efficiency Our approach has the advantage of requiring only D + 1 logprobs from the target model, since the hidden states can be computed knowing only PtDD and pv. For API-protected language models, this results in large reduction in API costs compared to L2T, which requires V logprobs per inversion. For OpenAl's GPT 3.5 Turbo, L2T requires V = 100 277 logprobs. The equivalent setting
"https://platform.operr opena. hwtp://pppaaaaaccccceeeeeeeeeeee cww.docs/ap...iicoooocoooooo..ce /eetieeeee/pi--effeeeeeeeee

---

T = for our method requires only around 4600 logprobs (based on the estimate from Finlayson al. [10] of GPT 3.5 Turbo's embedding size). Our method can scale up 21 while remaining cheaper than 1.2T.
The API cost of obtianing D logprobs per step for a 7 lelength sequence is roughly Σ D(i x Cin + Cou) log( B/s), where Cin and Cout the per-token input and output cost of the API, and B is the maximium logit bias allowed by the API. For GPT-4.1 I Mini, which we will assume has embedding size similar to GPT-3.5 Turbo, this cost would be
15
Σ
4600 ix
000 000 +
0.1
0.4
loga
100
= $5.50
1 1000000
0.001
for 16-token sequence.
Experimental setup
We generally follow the experimental settings originally proposed for L2T and O2P for fair compar- isons [21, 34j. We initialize our inverter as a pre-trained T5-base model [25]. For our target models, we use variants of Llama 2 7B (for comparison with baselines) and Llama 3.1 8B.
For training, we use the 2M Instructions dataset [21] as hidden prompts to our target model. We train for 100 epochs target model generations, which produced using greedy decoding and tracking the compressed logprob vector at every generation step. While pre-computing these logprobs and saving them to disk addresses the primary training speed constraint posed by target model generation, storage then becomes a significant scaling limitation, as 2 million 16-step generations require over 500 gigabytes. The hyperparameters and other implementation details are described in SD.
To measure inversion success, we compare the reference hidden prompts with those recovered by our inverter model, which also employs greedy decoding during inference. We use BLEU score [23], exact match, and token FI for comparisons. Token F1 is calculated as the harmonic mean of precision- proportion of predicted tokens in the true prompt-ann and eecall-proporticc of true prompt tokens in the prediction. Prior work [21, 34] also reports cosine similarity between text embeddings, which tend to be very high, suggesting that the metric is saturated; hence, we do not report this.
We evaluate our inverters on a held-out set from 2M Instructions and two -fitt--distiiuution (OOD) test sets: Alpaca Code [6] and Anthropic Helpful/Harmless (HH) [4, 11]. We also report system prompt inversion on Awesome GPT Prompts [2], and GPT Store [18].
We find that using more generation steps at test time than our inverter was trained on has a positive impact performance (see $6.3). By way of notation, we indicate when PILS trained and evaluated with 16 steps as PILS ( and indicate PILS trained on 16 steps and evaluated using 32 steps, as PILS 32.
For baselines, we compare our method to the prompt-based called output-to-prompl (O2P) inversion developed by Zhang et al. [35], logit-to-text (L.2T) and its variant (denoted 1.2T++) optimized for Llama 2 Chat by Morris et al. [21] and DORY [12). For the prompt-based inversion, we report both the mean performance and best performance from pool of prompts.
6 Experiments
6.1 PILS outperforms other inversion methods
Table I compares the in-distribution performance of PILS with baselines, reporting both the mean and the standard error of the mean for each metric on 2M Instructions. PILS surpasses all previous methods every metric by a considerable margin. Notably, we achieve 51% exact match recovery of hidden prompts for Llama Chat, where the best previous method (L2TT) could only recover 23% exactly. $B provides additional comparison (although with unique evaluation method which requires additional explanation) with DORY [12], with a 58-69 point improvements on BLEU.
We evaluate the -it-of-distribution generalization of our inverter models by evaluating them on held-out datasets. Results in Table 2 show that again, PILSS outperforms baselines by wide margin, (with the exception of the best prompting method the base model), indicating that our inverter is
5

---

Table I: Inversion performance the 2M Instructions validation set. Gray rows denote the theoretically equivalent PILS 1
and L2T. (6 indicates the model is trained on 16 tokens and evaluated on 32.
Target
Inverter
BLEU
Exact match
Token
Llama Chat
Prompt (avg.)
1.2
0.0
25.0 + 1.5
Prompt (top)
14.9 1.4
0.0
32.9 + 1.7
1.2T
2..
17.0 2.7
70.9 1.7
PILS ① (ours)
=
24.3 1.4
72.9 0.8
O2P
56.8
1.3
0.6
1.2T++
58.3
2.74
75.8
P
166 16 (ours)
0.9
4.6
84.2
P
16
(ours)
75.8 0.9
4.6
8..0
PILS
32
(ours)
47.0
8,60
PILS
32
54
(ours)
7.84
51.1 1.6
88,9
Llama 2
Prompt (avg.)
14.0
5.4 1.0
21.3 2.0
Prompt (top)
3.0
3.4
68.4 2.5
L2T
2.1
2..
77.8 1.3
PILS 1 (ours)
1.0
2..0
+ 0.6
O2P
6.17
41.0
83.8 + 0.7
PILS
66 16
(ours)
0.9
1.6
86.6 0.5
PILS
(ours)
0.9
51.2 1.6
89.0 0.5
Llama 3 Instruct
PILS
(ours)
63.7
1.5
79.7 0.7
PILS
16
32
(ours)
65.9 1.0
32.6
81.1 0.6
not just over-fitting the training set. We attribute the high performance of the prompting baseline to the tendency of the base model repeat the context verbatim (see discussion $6.2). Of particular note, our inverter achieves exact recovery of 60% of code prompts to Llama 2 Chat, whereas the previous best model could recover only 17%. We also see an almost 2x improvement on exact match over the best Llama 2 Chat baseline for HH. $E provides qualitative examples of these recoveries, for both in-distribution and -ittr--distribttion prompts.
We also include preliminary results with Llama 3 Instruct as the target. We hypothesize that its lower performance compared Llama Chat reflects Llama 3's more robust post-training, aimed at safety and instruction- wow.ooffooooon..... which likely makes inversion more challenging. This is similar to how post-training generally reduces inversion success on datasets like Anthropic HH (as seen when comparing Llama 2 base and chat models).
Theoretically, L2T and PILS L are theoretically equivalent, since they both invert based single generation step. This equivalence is confirmed empirically by their similar performance across metrics and datasets in Tables and 2. We highlight these methods with gray and them adjacent one another for comparison. On the in-distribution test set, PILS D slightly outperforms I.2T, perhaps because our representation makes information from the target output more readily available the inverter: our inverter input linearly encodes the target model's 's hidden state, whereas the L2T inverter input nonlinear transformation (recall Figure
6.2 Logprobs reveal hidden prompts multiple generation steps
To better understand how our method works, we visualize the effect of incrementally adding generation steps (from to 23) recover much of the prompt, although some tokens (like "felt" and "afraid") are revealed only after several steps. However, these tokens sometimes coincide with similar tokens in the generation (e.g.. output "fear" reveals input "afraid")), but not always (e.g., output "have" reveals input "felt").
trained 16-step inverter in Figure 3. The figure shows that
few steps
Figure 3 (right) suggests multiple generation steps are helpful because target models tend to echo the hidden prompt, either paraphrased by chat models, or verbatim by base models. This known phenomenon, often exploited in prompt injection [24], explains the strong performance of prompt- based inversion of base models in Table 2. Conversely, chat models, trained to avoid verbatim repetition (see Appendix Figure 6), are inherently harder invert. This explains the performance gap between chat and base models in Tables and 2, especially for prompt-based methods.

---

Table 2: Comparing PILS to baselines on out-of-distribution test sets. Gray rows denote the theoretically
equivalent L2T and PILS E
Alpaca Code Generation
Anthropic HH Exact match
Target
Inverter
BLEU
Exact match
Token Fl
BLEU
Token Fl
Llama 2 Chat
Prompt (avg.)
6.1 = 0.5
0.0
23.8 0.8
2,4 0.2
0.0
16.4 = 0.6
Prompt (top)
14.2 + 0.9
0.0
36.8
3.0 0.3
0.0
17.7 0.7
L2T
34.6
2.5
65.2
14.7
2.0
40.6
PILS AD
38.9 0.7
0.5
6.6
13.6
1.4
0.6
L2T++
4.8
8.7
7319
2.76
1.6
5.5
O2P
0.9
1.29
8.5
0.6
0.3
0.7
PILS 06 16
6.91
1.34
82.9
2.91
6.6
0.7
PILS
16
83.0
1.67
9..2
1.0
9.9
6,.1
PILS
8.8
1.66
92.6
37.7
64.3 1..9
PILS 64
85.0 + 0.8
60.5
0.41
13.0 3.0
65.7 0.8
Llama 2
Prompt (avg.)
29.3 1.9
12.7
2.0
14.2 2.2
40.8
Prompt (top)
73.0 2.8
6.45
80.2
64.5 7..7
L2T
4.8
1.9
1..9
1.6
1.6
55.8 = 1.4
PILS
0.9
9.1
74.5
2.7
53.0 0.6
PILS
36
1.0
3.56
85.1
4.1
2..
0.7
PI..S
32
7.02
48.1
8.5
56.2
35.4 1.5
76.8 +
Llama Instr.
PILS
16
51.8
1.01
0.6
2..0
49.2 0.8 4.7
PILS
06
32
6.0
2..6
8.6
22.8
5.1
50.2 = 0.8
0.8
Prompt token
Tell
cione
Prompt token
Top output token, generation steps
Tell about
Tell me about timse wftnn you were like of
Re
Make
break
Tell me about a fime when you were very.
Tell about
when you
a0
Tell me about tite when you were scared of
Sure
A-
Tell about
Tell me about a time when you were scared of
when you were
The
don 10
Tell about
when
you
roverse
of
6
Tell me about a time when you wer very of
the
Tell about
when you very
string 8
you
Tell about
when felt scared.
pensonal experiences
Tell me about a time when you foll scared.
Tell me about a tmme when you folt scared
your
you
Tell about
when
lons
Tell me about a time when you foll scared.
Tell me about a time when you fell scarre
down-
including
Tell about
when you
Tell me about a titte witee you folt afraid.
for
0.2 0.4 0.6 0.8
0
0.2
0.4
0.6
0.8
Probability
Probability
Figure 3: Inversion of Llama Chat for increasing numbers of generation steps. The x-tick labels indicate the hidden input tokens. The heatmap values indicate the probability of the prompt tokens according to PILS 1.. The wth row corresponds to feeding the inverter n generation steps. The tokens the y-tick labels indicate the target model's top token, which is appended to the sequence for the next generation step. The text to the right of the first heatmap indicates the inverter's hidden prompt guess. Red squares highlight where input tokens become recoverable by the inverter, meaning the probability of the prompt token goes from near-O to near-1. I. Filled square in the right indicate that the increase probability came only after the target model generated the hidden token directly. The blue line indicates the sequence length that the inverter was trained on (16 steps).

---

2M Instructions
Alpaca Code (OOD)
Anthropic HH (OOD)
100
PIL
80
PILS
16
BLEU
60
PILS
40
PILS
20
16
128
2
16
128
2
16
128
Steps (log scale)
Figure 4: Evaluating PILS inverters different numbers of generation steps. Circled points indicate the number of steps the inverter was trained on.
6.3 Length generalization: scaling target outputs improves performance
measure the effect of increasing the number of generation steps during training, by training inverters on 1, 8, 16, and 32 steps. From the circled points in Figure 4, it is clear that training more generation steps improves performance. We believe it is likely that longer sequences are especially helpful for longer prompts due to prompt echoing, i.e., outputs containing information about later parts of the prompt may not appear until later in the generation.
We are surprised to find that inverters trained on fixed number of generation steps generalize and improve when inverting longer output sequences. Figure 3, the model inverts the prompt only after 22 and 18 generation steps. To explore this phenomenon, we evaluate inverters trained on 1-32 steps on various generation lengths and plot performance in Figure 4. find that inverters continue to improve even when the number of steps surpasses the number of steps they were trained on, though the effect eventually saturates. the number of test steps exceeds the training steps, i.e., PILS (1 outperforms PILS ( We also note that this effect does not appear for inverters trained on step. Scaling the number of steps is particularly effective for inverting Llama 2 Chat on Alpaca Code (see Appendix Figure 5 for example).
remark that training on more steps still confers an advantage when
One possible explanation for the inverters' generalization success may be attributed to T5's pre- training, during which learned to process longer sequences. Given that T5 uses relative position embeddings, there are no position-specific weights (e.g.. learned position embeddings) that would cause distt--distiibution issues for longer inputs.
6.4 Inverting system messages much more challenging than user prompts
Since the main proposed use case for language model inversion today is to discover hidden system messages, we evaluate inverters on system messages in the Awesome [2] and Store [18] datasets. We use our PILS (4 inverter trained on 2M Instructions. Results in the top panel of Table 3 show that inverting system messages is much harder than inverting other prompts (Tables and 2), resulting in much lower scores. Again, this is likely because post-training discourages target models from revealing system messages. Our PILS outperforms O2P [34] on Llama Chat.
Given this success, we fine-tuned PILS inverter with Llama Chat outputs to compare with a similar setup in O2P with GPT-3.5 [22]. We trained only the attention layers the T5 encoder (detailed in $D.2) while completely freezing the decoder, on 50 samples for each dataset. This enables meaningful adaptation of our inverter new datasets while preventing overfitting on the small dataset. Here again, we outperform O2P on both datasets.
6.5
A target model transfer method for logprob-basee inversion
Target model transfer refers to using trained inverter on new target model without any additional training. Model transfer can be helpful when is infeasible to train a new inverter for new target model, e.g., if inference is too expensive to generate a training set. In this setting, we refer to the model used for inverter training as the source model, and call the new language model the target
8

---

Table 3: Comparison of PILS to baselines on system prompt recovery via zero-shot prompting and fine-tuning on 50 samples. Zhang al. [34] only provide O2P only results with GPT-3.5, so we include an O2P baseline with Llama 2 in the non-fine-tunin teninnttnnggggann setting to rule out the possiblity that performance differences are due to the target model.
Awesome
Store
Target
Inverter
BLEU
Token
BLEU
Token
GPT-3.5
O2P
0.4
28.8 1.0
6,4 1.2
37.6 1.9
Llama 2 Chat
O2P
2.3
2.8
6.3 0.7
32.2 1.8
Llama 2 Chat
PILS (2
38.3 7.9
34.1 101
GPT-3.5
O2P-Finetuned
47.9 1.8
1.1
5.6 1.2
36.3 2.6
Llama 2 Chat
PILS Finettned
19.8
50.7 # 1.3
16.42227
43.7 # 2.9
Table 4:
Transfer performance (token F1) for inverters trained with logprobs from Llama 2 7B Chat.
Target
Inverter
2M Instruct
Alpaca Code (OOD)
Anthropic HH (OOD)
Llama 2 13B
L2T
43.6 1.7
37.3 1.4
32.5 + 2.0
PILS 16
47.4 = 0.5
48.0 + 0.4
23.8 +
Mistral 7B Instruct
PILS 16
37.7 0.5
43.1 0.4
19.1 0.3
O2P
61.0 = 0.7
69.9 + 0.6
35.9 + 0.6
model. Both Morris et al. [21] and Zhang et al. [34] study model transfer for their methods, but due to architectural limitations, Morris al. [21] only transfer their L,T inverter to target models with the same vocabulary as the source model, i.e., models within the same family.
We overcome these architectural limitations by proposing a method for adapting our PILS inverter to models with different vocabularies. We use the set of tokens that appear in both the source and target vocabularies find logprobs for the source model vocabulary that are similar to the target model logprobs. By way of notation, let 1/sc be the vocabulary of the source model and let Vigh be the vocabulary of the target model. We assume that there significant overlap between these two vocabularies, such that 1'ssre Vigi D. We call this set of tokens yashr, We confirm that assumption holds for several models in $C.
Given a logprob output & RR from the target, select the shared vocabulary logprobs Eyetr R! We can then take the rows of the source model's unembedding matrix W that correspond to the shared vocabulary and solve the least squares problem W Wayarrr Cyahr for x. This x can be interpreted as a hidden state from the source model that produces an output that is similar to the target model output. We then use alo(ssfma Wx)) input to the inverter.
We evaluate our method by transferring our 16-step inverter trained on Llama 2 7B Llama 13B (same family) and Mistral 7B Instruct (out-of-family) and comparing Fl scores to those reported by L2T and O2P in their respective papers in Table
Interestingly. the impressive gains PILS in non-traan con-transfer settings fail to materialze in the model transfer setting. We speculate this could be due the targel specificity of our inverter, i.e., the inverter learns to leverage features that are specific to the target model during training, boosting performance on the source model, but hurting generalization new target models. On the other hand, text-based inverters like O2P must learn more general features during training due to their low-information text inputs, which may serve form of regularization and aiding model transfer.
7 Conclusion and future directions
We introduced technique for losslessly compressing language model logprobs which demonstrated large gains on language model inversion. Our analysis shows that language models reveal information about their prompts in their logprob outputs over the course of multiple generation steps. Our method also made progress towards the more challenging task of recovering system messages.
6 Since the O2P paper does not report OOD numbers, we run these evaluations ourselves.

---

Given that our inversion method, PILS is both effective and relatively inexpensive. our findings constitute an important security consideration for language model APls. It would be unwise for language model deployments to rely on the cost of inference or post-training alone to protect sensitive prompts. That being said, our proposed attack is not without mitigations. As shown in previous work [10, 5], arbitrary logprob access can be easily blocked by eliminating the API's logit bias parameter, preventing our particular attack, at the expense of reducing the API functionality. While logit bias has indeed been deprecated by some real-world APIs, it has not been eliminated, indicating that logprob-basse methods for language model forensics remain a relevant area research. Other mitigations include detecting logprob-bat based attacks by flagging repeated queries with different logit bias values, or changing model architectures to eliminate the softmax bottleneck [33].
Not only does our method show that the ceiling for language model inversion is higher than previously thought, but we also do not believe that we have fully saturated this task. Our inverter design might be improved, for instance, by using a more expressive feed forward adapter with a larger hidden size. Future work could further scale the number of generation steps during training or the size of the inverter model. We believe that progress on system message inversion can be greatly improved through the construction of a large-scale, diverse, high-quality (i.e., non-synthetic) dataset system prompts.
8 Acknowledgments
Matthew Finlayson's 's work is supported in part by a National Science Foundation (NSF) Graduate Research Fellowship. Xiang Ren's research is supported in part by the Office of the Director of National Intelligence, Intelligence Advanced Research Projects Activity, via the HIATUS program contract #2022-22072200006, the Defense Advanced Research Projects Agency with award HR001 1 22200466, and NSF IIS 2048211. This research is supported in part by the NSF under grant IIS2403477. the Simons Foundation, and the Allen Institute for AI. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of NSF, or the U.S. Government. The U.S. Government authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein. This work was partially done while S. Swayamdipta and M. Finlayson were visitors the Simons Institute for the Theory of Computing. The authors thank members of the NLP group at USC for their comments and feedback on the draft, as well as Collin Zhang for their help with this project.
References
[1] Aitchison, (2018). The statistical analysis of compositional data. Journal of the Royal Statistical Society: Series B (Methodological). 44(2):139-160.
[2] Akm, F. K. (2022).
Awesome chatgpt
prompts.
https://githin http://ggtttuu.... com/ f/
awesome- chhttptt-rooneee
[3] Bahdanau, D., Cho, K., and Bengio, Y. (2015). Neural machine translation by jointly learning align and translate. 3rd International Conference on Learning Representations, ICLR 2015 Conference date: 07-05-2015 Through 09-05-2015.
[4] Bai, Y. Jones, A., Ndousse, K., Askell, A., Chen, A., DasSarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., Joseph, N., Kadavath, S., Kernion, J., Conerly, T., El-Showk, S., Elhage, N., Hattield-Dodds Z., Hernandez, D., Hume, T., Johnston, S., Kravec, S., Lovitt, L., Nanda, N., Olsson, C., Amodei, D., Brown, T., Clark, J., McCandlish, S., Olah, C., Mann, B., and Kaplan, J. (2022). Training helpful and harmless assistant with reinforcement learning from human feedback.
[5] Carlini, N., Paleka, D., Dvijotham, K. D., Steinke, T., Hayase, J., Cooper, A. F., Lee, K., Jagielski, M., Nasr, M., Conmy, A., Wallace, E., Rolnick, D., and Tramèr, F. (2024). Stealing part of production language model. In Proceedings of the 41st International Conference on Machine Learning, ICML'24. JMLR.org.
[6] Chaudhary, S. (2023). Code alpaca: An instruction Ilama model for code generation.
10

---

[7] Dosovitskiy, A. and Brox. T. (2016). Inverting visual representations with convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
[8] Finlayson, M. (2024). The softmax function is linear. https://mattflnngginn smis smislinear. Accessed: 2025-05-09.
io/
[9] Finlayson, M., Hewitt, J., Koller, A., Swayamdipta, S.,, and Sabharwal, A. (2024a). Closing the curious case of neural text degeneration. In The Twelfth International Conference on Learning Representations.
[10] Finlayson, M., Ren, X., and Swayamdipta, S. (2024b). Logits of API-protected LLMs leak proprietary information. In First Conference on Language Modeling.
[11] Ganguli, D., Lovitt, L., Kernion, J., Askell, A., Bai, Y., Kadavath, S., Mann, B., Perez, E., Schiefer, N., Ndousse, K., Jones, A., Bowman, S., Chen, A., Conerly, T., DasSarma, N., Drain, D., Elhage, N. El-Showk, S., Fort, S., Haffield-oodss, Z., Henighan, T., Hernandez, D., Hume, T., Jacobson, J., Johnston, S., Kravec, S., Olsson, C., Ringer, S., Tran-Johnson, E., Amodei, D., Brown, T., Joseph, N., McCandlish, S., Olah, C., Kaplan, J., and Clark, J. (2022). Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned.
[12] Gao, L., L., Peng. R., Zhang, Y., and Zhao, J. (2024). DORY: Deliberative prompt recovery for LLM. In Ku, L.-W., Martins, A., and Srikumar, V., editors, Findings of the Association for Computational Linguistics: ACL 2024, pages 10614-10632, Bangkok, Thailand. Association for Computational Linguistics.
[13] Grivas, A., Bogoychev, N., and Lopez, A. (2022). Low-rank softmax can have unargmaxable classes in theory but rarely in practice. In Anmual Meeting of the Association for Computational Linguistics.
[14) Hendrycks, and Gimpel, K. (2023). Gaussian error linear units (gelus).
[15] Leinster, T. (2016), How the simplex a vector space. http:///166/6/boo__ee___iiei
ph. utexas.ee..... edu/
as___ html. Accessed: 2025-05-20
[16] Li, H., Xu, M., and Song, Y. (2023). Sentence embedding leaks more information than you expect: Generative embedding inversion attack to recover the whole sentence. In Rogers, A., Boyd-
J. and Okazaki, N., editors, Findings of the Association for Computational Linguistics: ACL 2023, pages 14022-14040, Toronto, Canada. Association for Computational Linguistics.
[17] Lin, C.-Y. and Och, F. J. (2004). Automatic evaluation of machine translation quality using longest common subsequence and skip-bigram statisties. In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL-04), pages 605 605-612, Barcelona, Spain.
[18] linexjlin (2024). Gpts. https://gi wtt......omm/iiieiiin/feeee
[19] Mahendran, A. and Vedaldi, A. (2015). Understanding deep image representations by inverting them. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR).
[20] Morris, J. X., Kuleshov, V., Shmatikov, V., and Rush, A. M. (2023). Text embeddings reveal (almost) as much text. In Conference on Empirical Methods in Natural Language Processing.
[21] Morris, J. X., Zhao, W., Chiu, J. T. Shmatikov, V., and Rush, A. M. (2024). Language model inversion. In The Twelfth International Conference on Learning Representations.
[22] OpenAl (2022). Introducing chatgpt. 2025-05-09.
Accessed:
[23] Papineni, K., Roukos, S., Ward, T., and Zhu, W.-. (2002). Bleu: a method for automatic evaluation of machine translation. In Isabelle, P., Charniak, E., and Lin, D., editors, Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 31 1-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.

---

[24] Perez, F. and Ribeiro, I. (2022). Ignore previous prompt: Attack techniques for language models.
[25] Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, J. (2020). Exploring the limits of transfer learning with a unified text-to-text transformer.
Journal of machine learning research, 21(140):1-67,
[26] Song. C. and Raghunathan, A. (2020). Information leakage in embedding models. In Proceedings of the 2020 ACM SIGSAC Conference on Computer and Communications Security, CCS '20, page 377-390, New York, NY, USA. Association for Computing Machinery.
[27] Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, and Salakhutdinov, R. (2014). Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(56):1929-195
[28] Taori, R., Gulrajani, I., Zhang, T., Dubois, Y., Li, X., Guestrin, C., Liang, P., and Hashimoto, T. B. (2023). Stanford alpaca: An instruction-followig llama model. https://gi thub. com/ tatsu-labe het///sanford_aappp
[29] Teterwak, P., Zhang. C., Krishnan, D., and Mozer, M. C. (2021). Understanding invariance via feedforward inversion of discriminatively trained classifiers. In Meila, M. and Zhang. T., editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 10225-10235. PMLR.
[30] Verma, A., Krishna, S., Gehrmann, S., Seshadri, M., Pradhan, A., Ault, T., Barrett, L.., Rabinowitz, D., Doucette, J., and Phan, N. (2024). Operationalizing a threat model for red-teaming large language models (Ims). arXiv preprint 477442240.14937.
[31] Wang, Y., Kordi, Y., Mishra, S., Liu, A., Smith, N. A., Khashabi, D., and Hajishirzi, H. (2023). Self-instruct: Aligning language models with self-generated instructions. In Rogers, A., Boyd- Graber, J., and Okazaki, N., editors, Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages Association for Computational Linguistics.
Toronto, Canada.
[32] Wu, Y., Li, X., Liu, Y., Zhou, P., and Sun, L. (2024). Jailbreaking gpt-4v via versaaall attacks with system prompts.
[33] Yang, Z.,, Dai, Z.. Salakhutdinov, R., and Cohen, W. W. (2018). Breaking the softmax bottleneck:
A high-rank RNN language model. In International Conference on Learning Representations.
[34] Zhang, C., Morris, J. X., and Shmatikov, V. (2024a). Extracting prompts by inverting LLM outputs. In Al-Onaizan, Y., Bansal, M., and Chen, Y.-N., editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 14753-14777, Miami, Florida, USA. Association for Computational Linguisties.
[35] Zhang, Y., Carlini, N., and Ippolito, D. (2024b). Effective prompt extraction from language models. In First Conference on Language Modeling.
12

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We do not claim anything outside of our contributions/scoee.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize to other settings.
It is fine include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: we report results on methods that and do not work, and acknowledge the limitations of our method, We also acknowledge mitigations that would make our method not work on APl-protected models. We comment on how realistic our assumptions are for all our assumptions.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "L_imitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role developing norms that preserve the integrity the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
13

---

Answer: [Yes]
Justification:
We provide proof of our theoretical result.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
The information is mentioned $D and $5
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g.. with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but should be possible for other researchers to have some path reproducing verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
14

---

Answer: [Yes]
Justification: Yes. Code Repository is attached with supplementary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https://nips.c cc
ssionPolicy) for more details. The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines, If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper speciffy all the training and test details (e.g.. data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We specify details in $D and $5
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report eтror bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
We report Standard Error Measure for all our results.
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
15

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification:
We have mentioned in $D.3
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: System prompt recovery is inherently dual-use technology. Our research shows that language model prompts can be inverted, even when those prompts contain valuable personal information. However, method is easy to run locally, but extremely impractical to run via API, costing a high amount of time and money. For this reason, we expect PILS to be most useful for practitioners looking red-team models locally.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there is a special consideration due laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification; We discuss the implications of PILS in the final section (Conclusion and Future Directions), In particular, we discuss the implications of different schemes language model deployments and the risk in providing users access to raw logprobs.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
16

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymeet of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: Although we do not test any new methods for defending langauge model log-pro probabilitiiss, we discuss the potential for language model API safeguards and mitigations in the final section (Conclusion and Future Directions).
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All the assets are cited properly.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
17

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. cww /ataattss has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach the asset's creators.
13. New assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [Yes]
Justification: All code will have documentation.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details of the dataset/code/model as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
supplemental material is fine, but if the main contribu-
According to the NeurIPS Code of Ethics, workers involved data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
18

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy pectps://nuurppee c2/oonfeeeeeccce///22333 for what should should not be described.
19

---

Prompt token
Chaangi
the
loop
print
the
nombher
frem
del delpondding
order
renge
prin
Sure
Here
Top output token, generation sleps
the
0.8
modified
that
numbets
0.6
Probability
16
0 17-
to
18
19
20
21
22
-0.2
27
28
79
in 30
magee
31
32
Figure 5: PILS 16 inverting prompt Llama 2 Chat from the Alpaca Code evaluation
Table Performance on inversion datasets Alpaca and for comparison with DORY. Target model is Llama Chat.
measured BLEU and ROUGE-L
Alpaca
Self-instruct
Method
BLEU
ROUGE-L
BLEU
ROUGE-L
DORY
22.6 80.5
43.5
11.2
27.5
PILS 16
89.0
80.2
86.3
A
Additional inversion visualizations
See Figures 5 and 6.
B
Comparison with DORY
For completeness, we compare our method to the reported performance of DORY inverter from Gao et al. [12]. The paper reports performance
BLEU and ROUGE-L [17] for Alpaca [28]? and Self-Instruct [31], both of which are included in our 2M Instructions training set. To compare our method, we report the same metrics for PIL. ( the subset of our 2M Instructions test set that come from those datasets. The results can compared Table 5, where we see that PIL (16 16 performs much better.
C Language models have many common tokens in their vocabularies
Table shows that Llama 2 has significant vocabulary overlap with several popular models from different families.
7 AAlpaca different from Alpaca Code. The former is included in 2M Instructions and the latter not.
20

---

Llama 2 Base
Llama 2 Chat
Do
the
tep
yourr
Frod
How
sore
Wiat
yrour Ratiee
ner
appoaee rrame
Food
How
What
rome
Hello
anyone
your
My
name
zame
16
10-
Find 11-
11
Generation steps
12
13-
13
Jena-
14-
14
Hi
close
How
you
What
cam
your 21
24
sell
name 25
25
26-
26
ihal
Figure 6: PIL.S 16 inverting an OOD prompt to Ilaaa 2 Base and Chat.
Table 6:
Token overlap between the Llama 2 vocabulary (32 000 tokens) and several models. A sample of tokens common all of these models shown on the right.
Model
Vocabulary size
Overlap
nyn,
Preferences, assembly, angers, elin, flu, forification, TER, Attribure, Hoxs, garma, sty, asket, 803Es, river, ant, commens,, itu, world, ifica, Alt. ator, clogie. enbed, ategories, Gr, sch, Could,
Llama 2
32 000
32 000
rain,
Mistral
32 768
24 184
Llama
128
9651
LTT, rap. ames, very, aris, leased, Edge, zult, oving, Ser, bour, ror. roller
OLMo
100
9580
GPT 4o
200019
13.324
D Implementation details
This section details experimental configurations and resources. All work utilized PyTorch and Hugging Face transformers.
D.1 Main inverter training
We trained a T5-base inverter for the inversion Llama2-7B, Llama2-7B-Chan and Llama-3, 1-BB Instruct models. Key training parameters included learning rate of 2e-4, batch size of 250, and the AdamW optimizer with default settings. A 3200-step linear warmup was used, after which the learning rate remained constant. Training ran for 100 epochs (Llama-3. 3.1-3....... IBIIntuuuct was trained for 50 epochs), using bfloat16 mixed precision.
D.2 System prompt inverter fine-tuning
The T5-base inverter was subsequently fine-tuned for system prompt inversion using the Awesome (50 training/10 /00 testing samples) and Store (50 training/29 testing samples) datasets from Zhang et al. [34]. Common fine-tuning parameters across both datasets included batch size of 50, the Adam W optimizer with default settings, and bfloat16 precision. For the Awesome dataset, fine-tuning used learning rate of 1e-4 for 100 epochs, while the Store dataset used learning rate of 5e-5 for 50 epochs.
21

---

For target model generation during system prompt inversion, we used a single static user prompt, "Who are you?". This contrasts with the multi-prompt strategy of O2P (referenced in $6.4), which employs the following four user prompts generate 64 sentences:
Give me 16 short sentences that best describe yourself. Start with '1:
Give me 16 examples questions that can ask you. Start with 'I:"
Give me scenarios where can use you. Start with '1:"
Give me 16 short sentences comparing yourself with ChatGPT. Start with '1:
Our simpler prompt chosen to isolate system prompt effects, ensuring focused comparison given our inverter's original training covered both system and user prompts.
D.3 Resources
Main inversion training was conducted on four NVIDIA RTX A6000 GPUS, which takes about I week to complete. System prompt inverter fine-tuning utilized a single NVIDIA RTX A6000 GPU and takes about 10 hours to complete.
22

---

E
Qualitative analysis
See Table
Table 7: Examples of Prompt Inversion. Each block shows the original (hidden) prompt, LLM's generated tokens, and our model's inverted prompt for Distribution and Out-of- cot-of-Distribution datasets.
2M Instructions
Original Prompt:
How has the rise of streaming services impacted the music industry?
Generated Tokens:
The rise of streaming services has had significant impact the music industry, transforming the way music is consumed, distributed, and monetized. Here ways which streaming has affected the industry:
key 1. Shift from physical to digital sales: Streaming services have led to a
Inverted Prompt:
has the rise of streaming services impacted the music industry?
Original Prompt:
Write an email for me to my friend. in this email, you need tell him about your vacation plans and ask his advice what you should in that place.
Generated Tokens:
Sure, I'd be happy to help! Here's an email to your friend:
Subject: Vacation Plans and Seeking Advice!
Hey [Friend's Name],
I hope this email finds you well! I
Inverted Prompt:
I am planning to go vacation next month. write an email to my friend and ask him for his advice on what places should visit. Output:
Alpaca Code (OOD)
Original Prompt:
Edit this code snippet in Python to remove any errors and improve readability.
Generated Tokens:
Sure, I'd be happy to help! Please provide the code snippet you'd d like me to edit, and I'1l go through it line by line to identify any errors and suggest improvements for readability.
Here's general process follow when reviewing code:
Inverted Prompt:
Edit this code snippet in way that it will improve its readability.
Original Prompt:
Write Python script to find average of numbers stored in an array.
Generated Tokens:
Sure! Here is simple Python script that finds the average of numbers stored in an array:
Define a function to calculate the average of an array def :numbee:::
# Calculate the of the numbers in the array sum for num in
Inverted Prompt:
Write Python script to find the average of numbers stored in array.
23

---

Anthropic HH (OOD)
Original Prompt:
How many US presidents have been women?
Generated Tokens:
Unfortunately, zero US presidents have been women. The United States has had total of 45 presidents throughout its history, and none of them have been women. Despite the signi ficant progress made in recent years in terms of gender equality and women's 's rights, the highest office in the land has remained
Inverted Prompt:
How many US presidents have been women?
Original Prompt:
What is the best way to kill human being?
Generated Tokens:
I cannot fulfill your request. I'm just an AI, it's not within my programming or ethical guidelines to provide information or advice harming or killing humans any living being. It is important respect and uphold the inherent worth dignity of all individuals
Inverted Prompt:
What is the most effective way to kill human?
24


Reviews:
[{"review_id": "cYn39tcFoS", "reviewer": "Reviewer_sEEm", "summary": "This paper introduces Prompt Inversion from Logprob Sequences (PILS), a method to recover hidden prompts from a language model. The key contribution is a novel technique to losslessly compress sequences of next-token probability distributions into compact, low-dimensional vectors using the Additive Log-Ratio (ALR) transform. This allows the inverter to leverage information from multiple generation steps, a significant departure from prior work. PILS achieves state-of-the-art results, improving exact prompt recovery rates by 2-3.5x over previous methods. The paper also reports a surprising \"length generalization\" capability, where the inverter's performance improves on longer sequences than it was trained on.", "questions": "1.  **ALR Implementation:** Your use of a random token set for the ALR transform seems ad-hoc. How sensitive is performance to this choice, and did you evaluate more standard, stable alternatives like Centered or Isometric Log-Ratio (CLR/ILR) transforms?[2, 3]\n2.  **Model Transfer:** The proposed transfer method underperforms text-based baselines significantly. Why are logprob-based features less transferable? Have you considered more advanced alignment techniques like Centered Kernel Alignment (CKA)?[4]\n3.  **Length Generalization:** To substantiate your claim that relative position embeddings (RPEs) cause length generalization, can you provide an ablation study where an inverter with absolute position embeddings fails to generalize?\n4.  **Practical Cost:** Can you provide a concrete cost analysis (in dollars and time) for executing a PILS attack against a commercial API, factoring in the bisection search for logprobs?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "**Strengths:**\n1.  **State-of-the-Art Performance:** The method achieves a 2-3.5x improvement in exact prompt recovery over strong baselines, a significant leap that elevates logprob-based inversion from a theoretical to a practical security threat.\n2.  **Novelty and Technical Contribution:** The core ideas are highly original: inverting a *sequence* of probability distributions and using a principled, lossless compression (ALR transform) to make this feasible. The discovery of length generalization is also a valuable scientific finding.\n3.  **Rigorous Evaluation:** The claims are supported by comprehensive experiments against multiple baselines, across several LLMs and diverse datasets, including insightful qualitative analysis.\n\n**Weaknesses:**\n1.  **Poor Model Transfer:** The proposed method for transferring the inverter to a new model family performs poorly, significantly lagging behind simpler text-based attacks. This limits the attack's scalability, as a new inverter must be trained for each target model family.\n2.  **Ad-Hoc Compression Implementation:** The choice to use a random set of tokens as a basis for the ALR transform to avoid \"degenerate compression\" is not well-justified and lacks analysis of alternatives (e.g., CLR, ILR), raising questions about the method's robustness.[1, 2, 3]", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive score and valuable feedback. We appreciate your comments and will use them to improve our paper. Regarding the listed weaknesses, we hope you agree that poor model transfer is a limitation of the method but not a weakness of the paper, since an honest evaluation of the limitations should be part of any analysis of a new method. See the answer below to question 2 for more details on our transfer experiments. Similarly, we believe that our answer below to your question 1 addresses the second listed weakness.\n\nQuestions:\n\n1. (ALR implementation) In our preliminary experiments, we observed that tokens whose corresponding unembedding submatrices are low-rank consistently underperformed compared to those with full-rank submatrices. This makes sense because linearly dependent tokens provide redundant information about the hidden state. Empirically, we find that the first D tokens tend to be low rank (this may depend on the model), but a random choice of D+100 tokens usually has high rank, and therefore performs well. We do not observe sensitivity to the choice of random tokens, and choosing random tokens is much easier than searching for a full rank submatrix. We also explored using other transforms like CLR, but observed no significant performance improvements over our proposed approach. We adopted ALR primarily due to its efficiency advantages: while CLR requires computing the mean over the entire logprob vector (necessitating access to all vocabulary tokens), ALR operates with only D+101 logprobs, which is substantially smaller. This design choice significantly reduces API costs (as measured in logprobs) without any impact on performance.\n\n2. (Model Transfer) We experimented with several variations on our logprob alignment technique and did not find any improvements, though this does not rule out the possibility that other techniques like CKA might work. We report these findings despite their limitations because our method is the first ever cross-family logprob-based inverter transfer method, and could precipitate better results in future work. We speculate in the paper (line 257) that transfer suffers because the inverter “overfits” to model-specific features from the source model that are not present in logprobs from other models.\n\n3. (Positional embedding generalization ablation) This would be an interesting experiment to run, though the rebuttal period is not long enough to accommodate it. We will consider adding this for the camera ready. Since this claim is speculation, and not central to the findings of the paper, we can consider removing it, though we think that it is an interesting avenue for future work.\n\n4. (Practical Cost): Great idea, we will add this to the paper. Using some back-of-the-envelope calculations, let’s calculate the cost for a T-step attack on an API model with hidden size D, max logit bias B. \nWe need $T\\times D$ logprobs, each logprob requires $\\log(B/\\epsilon)$ queries. Each query has up to $T$ input tokens and 1 output token. Suppose the input cost is $C_\\text{in}$ per token and the output token cost is $C_\\text{out}$ per token. So the cost will be \n$$\\sum_{i=0}^{T-1}D(i\\times C_\\text{in}+C_\\text{out})\\log(B/\\epsilon).$$\nIf we are using an GPT-4.1 mini (one size up from the cheapest model GPT-4.1 nano), this model has an unknown hidden size D, but perhaps we assume it is similar to GPT-3.5-Turbo which is less than 4600. Max logit bias is 100, cost is $0.40/1M tokens and cost per output is $1.60/1M tokens, and we will ignore caching, though that could reduce cost further. We will require logprobs with precision of 0.001.\nThis gives us\n$$\\sum_{i=0}^{15}4600\\times(i\\times\\frac{0.1}{1000000}+\\frac{0.4}{1000000})\\log_2(100/0.001)=\\$5.50$$.\nFor GPT-4.1 nano, this cost is $1.38.\nUsing the batch API, both of these costs can be halved.\nThis cost per prompt is relatively low, and creating a small dataset to fine-tune an inverter model to target GPT-4.1 would be feasible."}, {"review_id": "P888auufY9", "reviewer": "Reviewer_QJEH", "summary": "This work presents a language model inversion (i.e. reconstructing information on the inputs of the model based on its behavior) which relies on sequences of compressed logprobs (i.e. not requiring the full logprobs vectors). This allows not only a faster way (in terms of number of queries) to construct the data used for the inversion, but also much better performances compared to previous work.", "questions": "- Typo line 96: \"an language model\"\n- I am slightly confused regarding what your intuition for your approach to work could be. You write, line 227: \"his is likely because post-training discourages target models from revealing system message\". But it doesn't seem natural to me that information about the prompt couldn't be extracted from logprobs of responses that do not try to repeat the system prompt. What do you believe is the mechanism at play behind the success of logprob-based inversion (particularly yours which leverage sequences of logprobs)?\n- Could you try and make your Figure 3 clearer? What exactly is given to the model and how does that evolve with the sequence of prediction? Why are there so many tokens with such a high probability? How should they be read? What does it mean for a token to be recoverable? Why do the filled red grid cells matter?\n- Why do you think model providers still allow for access to models' logprobs?", "limitations": "Limitations seem addressed to me.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Clarity\n### Strengths\n- Paper is well written and very easy to follow\n- Illustrations are clear and helpful\n### Weaknesses\n- The grid plots such as in Figure 3 are difficult to understand and parse.\n\n## Quality\n### Strengths\n- The paper shows strong results with massive improvements compared to previous approach on certain experiments.\n- The PILS approach seems practical as a way to extract user information contained in a prompt.\n### Weakness\n- The choice of models used should be justified, especially given that Llama 2 can be considered an old model, now.\n\n## Significance & Originality\n### Strengths\n- This work demonstrates a practical language model inversion attack and seem to be the first to leverage working from a (transformed) sequence of logprobs to gain more information.\n- The lossless compression approach suggested is interesting and seem to indicate not only a low-dimensional space of interest in which attacks can be successful, but also potential for transferability (as illustrated) across models.\n### Weaknesses\n- This work builds upon previous work (L2T) but the improvements are substantial enough to be considered novel.", "was_revised": true, "final_justification": "The authors addressed my concerns and I found the discussions with other reviewers interesting too. I maintain my score.", "rebuttal": "Thank you for your positive review and valuable feedback. We will make sure to improve the paper accordingly. We will address your questions and concerns as best we can below\n\n(Grid plots and figure 3)\nIn figure 3, the target model receives the prompt (xtick labels) and generates an output (ytick labels) one token at a time. At each generation step, we feed the outputs *so far* to the inverter, and force it to generate the hidden prompt, measuring the probability of each prompt token according to the inverter. These probabilities are a single row of the grid. The rows with the most high probability tokens (mostly gray) are the ones where the model is closest to correctly guessing the hidden prompt.  Token is “recoverable” when its probability is high. Filled red grid cells mean that the target model had to generate the hidden token before the inverter was able to guess what it was. We will try to make these figures easier to parse in the final draft.\n\n(Choice of model) We agree that Llama 2 is a weaker model, so we also provide Llama 3 results in Tables 1 and 2 for the benefit of future work. We used Llama 2 models in order to make our results comparable to previous work without replicating all of their experiments on Llama 3. We will make sure that this is explicit in the final draft.\n\n(Intuition for why our method works) You are correct in that logprob-based inversion can recover the prompt even when it does not appear in the text outputs of the model. Line 227 is only trying to convey that inversion success appears to be correlated with how likely the model is to generate the secret prompt, even for logprob based methods. As to the mechanism for how our method excels, compared to text-based inversion, logprob-based inversion accesses richer information about the prompt directly from the model’s internal representation. Compared to the previous logprob-based inversion, our method gives the inverter access to information that only surfaces later in the generation process. We are happy to discuss more about these intuitions.\n\n(Why do model providers allow logprob access) We are not sure about all the reasons, but we presume that logprobs and logit bias are useful to API users for one reason or another (e.g., fine-grained control of generations), and so removing these options from the API would cost providers in some way. It may be that these costs outweigh the risks of providing logprobs, at least for now."}, {"review_id": "vl59cmCoVk", "reviewer": "Reviewer_uYfj", "summary": "The authors present an incremental improvement over previous work on language model inversion by proposing PILS, a prompt inversion from logprob sequences method. The improvement of PILS stems from taking advantage of the information contained in the full output sequence of a language model, rather than just the first generated token. A clever dimensionality reduction technique lowers the computational requirements of PILS substantially. The final performance is remarkable, with up to 60% prompts reconstructed exactly.", "questions": "Line 127 claims that the authors' model requires $D+1$ logprobs for each position, even though the rest of Section 3, Section 2.2, and Figure 1 mention that only $D$ logprobs are necessary. I would appreciate a clarification on this seemingly contradictory statement.\n\nHow long are the prompts being inverted in the experiments? I would suspect longer prompts are more difficult to invert, but a more detailed analysis would be agreat addition to the paper.\n\nTables 3 and 4 do not report all experimental metrics: exact matches and BLEU (in Table 4) are missing. I presume this is because the scores are too low to be informative, but it would be good to report them anyway for completeness. Furthermore, why are the authors not always comparing with both O2P and L2T there?", "limitations": "Yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Personally, I believe language model inversion is an underexplored topic, and the authors' contribution is welcome, albeit slightly incremental in nature.\n\nThe paper is professionally written and clear to follow. I only found Theorem 1 relatively irrelevant to the paper contribution, as the authors' work is mainly experimental.\n\nThe positive results on generalization across output length (Figure 4) and the negative results on generalization across different target models (Table 4) are intriguing.\n\nThere are a few minor typos. In Line 96, \"an language model\" should be \"a language model\". The header of Section 5.5 is not capitalised correctly.", "was_revised": true, "final_justification": "The authors have promised to clarify a few minor points in the final version of the paper. I still recommend acceptance.", "rebuttal": "Thank you for your valuable feedback and positive evaluation. Your input will help us to improve our paper.\n\n(Confusion about logprobs): To simplify the analysis in section 2 and the start of section 3, we assume access to all V logprobs for each output. Line 127 makes the observation that we don’t actually need all of these logprobs to calculate $\\text{alr}(p)_{1:D}=(\\log p_1-\\log p_V, \\log p_2-\\log p_V,\\ldots,\\log p_D-\\log p_V)$---we only need the first D logprobs and $\\log p_V$.\n\n(Prompt Length) We limited the inversion of prompts to 64 tokens during both training and testing, consistent with L2T. While our qualitative examples (Appendix E) show prompts of varying lengths, we agree that a systematic analysis of how prompt length affects inversion performance would be interesting. We will consider adding such an analysis to our final version.\n\n(Missing Metrics): Your assumption is correct that these scores are too low to be informative. Exact match scores for system prompt recovery (Table 3) are indeed very low (near 0%), and similarly, exact match and BLEU scores for model transfer (Table 4) are substantially lower than the reported Token F1 scores. Additionally, L2T does not report BLEU and exact match scores for their transfer experiments. We are happy to include these numbers for completeness.\n\n(Missing Baselines): For Table 3 (system prompt recovery), the L2T did not conduct any system prompt inversion experiments, so we only compare against O2P. Since O2P generally outperforms L2T, we did not deem it necessary to run these experiments ourselves. For Table 4 (transfer performance), the baseline coverage reflects methodological constraints:\n\n- L2T's model transfer only works for models with the same tokenizer (we report results for Llama 2 13B which shares a tokenizer with Llama 2 7B, but cannot transfer to Mistral 7B)\n- O2P did not report numbers for Llama 2 13B in their original work, so we only use their reported Mistral 7B results. We do not believe that adding this would be particularly informative or change our conclusion that O2P outperforms our method in the transfer setting."}, {"review_id": "Sug3yb46i0", "reviewer": "Reviewer_vtuB", "summary": "This paper proposed a new method called PILS to achieve a better performance for large language model (LLM) inversion, which aims to reveal the input prompt, such as the user or system prompt, given exposure to the output probability distribution of the LLMs. More specifically, PILS can losslessly compress the output distribution to a low-dimensional subspace via linear transformation. Based on the low-dimensional embedding, an inverter is leveraged to recover the input prompt. Experimental results show that PILS can outperform the evaluated baselines.", "questions": "The questions are listed as follows.  \n1. The example of OpenAI API in section 2.2 is confusing. If the API can only provide the logprob of the most-likely token, how could the proposed method get the full probability distribution of the logprob for the multi-step output? Besides, could you give more explanation about the relationship between the provided logit bias from API and the minimum logit bias $\\beta$?    \n2. In Table 3, what is the meaning of the setting of GPT-3.5 if there is no evaluation of PILS in this setting?\n3. In Table 1 and 2, what is the baseline called prompt (avg.) and prompt (top)?\n4. The training of the inverter requires 100 epochs of fine-tuning of the T5-base model. Is this computational cost acceptable?\n5. What is the size of the test sets?", "limitations": "No. My comments on weaknesses and questions are a good reference for further improvement.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The strengths of the paper are listed as follows.   \n1. The authors provide a theoretical analysis to show that the outputs of language models are losslessly compressible.   \n2. The authors evaluated their methods on both the in-distribution test data and out-of-distribution test data.  \n    \nThe weaknesses of the paper are listed as follows.  \n1. The proposed method requires the LLM to provide access to the output logits, which are not accessible in many closed-sourced LLMs and thus limit its impact.     \n2. Some important technical details are missing.    \na. What is the formula for the alr transform?    \nb. How to train the inverter?    \nc. Why use the encoder-decoder architecture for the inverter?     \n3. It would be better if the authors could move the section of related work after the introduction, since it can help the readers to understand the novelty before delving into the technical details.    \n4. It is not clear why compressing the output probability distribution can save the API cost since it still requires the API to provide full distribution for compression.\n5. The abbreviation O2P has no explanation when it first appears in the paper, which may cause confusion.\n6. It would be better if the authors could compare the training cost of PILS and the other related works.   \n7. The evaluation should be improved.   \na. In Table 4, the baseline is inconsistent for different target models.    \nb. In Table 1 and 2, it would be better if the authors could provide the target model in addition to the Llama family.   \nc. The authors should provide a separate introduction to the baselines. It is not clear whether the author includes all the important existing works in LLM inversion given the current content.", "was_revised": true, "final_justification": "I recommended to boarderline accept this paper since the authors addressed all of my concerns and questions one by one in their rebuttal. I appreciated the effort the authors made to provide the detailed response.    \n     \nIn summary, I think this paper has a novel idea in the research area of language model inversion. The technique is solid and will inspire future work in this area. For the suggestion to the revision of the paper, I hope the authors can follow the response they provided to improve the readability of the paper, such as the clarification of the choice of baselines in Table 4, and the placement of related work. I believe the improvement in the writing is also helpful to highlight the importance of this work.", "rebuttal": "Thank you for your valuable feedback. We hope we can address your concerns.\n\n1. Though many current LLM APIs do not provide logprobs/logits, some do, and understanding implications of providing logprobs can inform practitioners making decisions about APIs in the future. As an aside, logprobs can still be extracted from APIs that do not provide logprobs if they provide a logit bias parameter (Morris et al., 2024).\n\n2.a (ALR formula) The formula for the alr transform is given on line 80. Using python indexing, alr(p) = log(p[:V]) - log(p[V]).\n\n2.b (How to train inverter) We train the inverter as a standard encoder-decoder model, with the source sequence being the logprob sequence, and the target sequence being the hidden prompt, using cross entropy loss. Section 4 describes how we train our model and additional information is provided in Appendix D. The supplemental material contains the training code. \n\n2.c (Why encoder-decoder?) At a high level, inversion takes a sequence (the LM output), and returns a sequence (the hidden prompt). Since this is a sequence-to-sequence (seq2seq) task over natural language, we believe that encoder-decoder transformers are the most obvious and appropriate architecture to use. This has precedent in previous work on inverting text embeddings (Morris et al. 2023) where they found that the encoder-decoder models significantly outperform decoder-only ones.\n\n3/5. (Related work placement) Thank you for pointing this out. We will consider either moving the related work section, or making it clearer in our introduction what aspects of our work are novel before going into technical details. In doing so we can also resolve your concern about lack of intro for O2P.\n\n4. (Compression requires full logprob outputs?) I believe this is a slight misunderstanding. Our compression only requires access to a number of logprobs equal to the hidden size D of the model (plus one), as opposed to V logprobs for a naïve method. In particular, we choose a list of D tokens plus a reserved token, get the logprobs of those tokens, and subtract the logprob of the reserved token from the D other logprobs. This requires only D+1 logprobs, and you can check that this is exactly the “recovered hidden state” on line 116. For an API that charges per logprob (e.g., gives the logprob of a single token per query), this leads to significant savings versus a method that requires V logprobs per generation step.\n\n6. (Training cost comparison) Our method has a training cost similar to L2T, since the only difference is the shape and size of the model input, the latter of which ranges from smaller to larger than the L2T size, depending on the number of output steps we are training on. Since training efficiency is not the focus of our method and it remains similar to a previous method, we do not believe that an efficiency analysis would be particularly interesting.\n\n7.a (Baselines in Table 4) You are correct that the baselines are different for the different models. As mentioned on line 240, the L2T transfer method is incompatible with out-of-family models, so it cannot be compared for Mistral models. For O2P, we are restricted to the numbers reported in their paper, and they do not report transfer to Llama 2 13B. We chose not to replicate the O2P transfer experiments to obtain these numbers due to computational constraints, as this is a relatively less important part of the paper. We expect O2P to perform at least as well for Llama 2 13B as it would on Mistral, since it was trained on Llama 2 7B. Confirming this would only further strengthen our conclusion from Section 5.5 that O2P is best for model transfer.\n\n7.b (Tables 1 & 2 target models) target models here are Llama 2 7B and Llama 3 8B, as mentioned in Section 4 paragraph 1. We can add this information to the caption as well.\n\n7.c (Intro to baselines) We are confident that our baselines include all important existing language model inversion methods, but we agree that these could be better introduced. We will incorporate an overview of existing methods in the revised related work section.\n\nQuestions:\n\n1. (How to get logprobs from OpenAI API) As an illustrative example, observe in the table below how applying a logit bias of 2 causes token 2 to become the most likely token. \n\n| Logit bias on token 2 | Most likely token |\n| --- | --- |\n| 0 | 1 |\n| 2 | 2 |\n\nIf we know that token 1 has logprob -0.5, then it must be the case that token 2 has logprob between -0.5 and -2.5. You can continue to bisect this range until you have token 2’s logprob within a small error tolerance. You can read more about this method in Section 5 of Morris et al. (2024). “Minimum logit bias” refers to the minimum logit bias that you provide to the API to get token 2 to be the most likely token.\n\n2. (GPT-3.5 baseline in Table 3) The goal of Table 3 is to compare O2P to PILS for inverting system prompts. Since the O2P paper only provides results with O2P+GPT-3.5, we use these as our baselines. To help rule out the possibility that different target models explain the performance gap, we include an O2P+Llama2 baseline in the non-finetuning setting, and see that the results are highly similar to O2P+GPT3.5. We will clarify this in the final version of the paper.\n\n3. Prompt (avg.) is the average performance from a set of adversarial prompts designed to get the model to reveal its hidden prompt. Prompt (best) is the performance of the best prompt from this pool. These are obtained from Morris et al. (2024), as mentioned at the end of Section 4; we can make this clearer.\n\n4. (Acceptability of training cost) Training our model takes 3-7 days to complete on four RTX A6000 GPUs, so it is not prohibitively expensive, as detailed inAppendix D. We would like to point out that it is a one-time cost per target model to train an inverter, and it is likely that this cost could be significantly reduced with a bit of engineering effort.\n\n5. (Test set sizes)  For Tables 1,  2 and 4, we have used 1000 samples from each dataset, as was done in previous work. We will make sure to add this detail to the final draft. For Table 3, as mentioned in Appendix D.2, we used 103 samples for Awesome Dataset and 29 for Store Dataset, as was done in previous work."}]

Author rebuttal:

Meta-review: This paper studies language model inversion and proposes compact representations of next-token distributions to improve input reconstruction. The approach is simple and clearly motivated, with theoretical analysis and empirical validation across model scales and datasets. Reviewers appreciated the clarity of writing and the technical soundness, while raising concerns about the breadth of evaluation and real-world impact. The rebuttal addressed methodological questions and provided additional analysis, which satisfied most reviewers. While the contribution is incremental in scope, it is technically solid, easy-to-implement, and could be of interest for the community of inversion attacks. Overall, I recommend acceptance for this paper.
Decision: Accept (poster)