Sentence Embedding Leaks More Information than You Expect: Generative Embedding Inversion Attack to Recover the Whole Sentence
Year: 2023

Abstract:
Sentence-level representations are beneficial for various natural language processing tasks. It is commonly believed that vector representations can capture rich linguistic properties. Currently, large language models (LMs) achieve state-of-the-art performance on sentence embedding. However, some recent works suggest that vector representations from LMs can cause information leakage (Song and Raghunathan, 2020; Pan et al., 2020). In this work, we further investigate the information leakage issue and propose a generative embedding inversion attack (GEIA) that aims to reconstruct input sequences based only on their sentence embeddings. Given the black-box access to a language model, we treat sentence embeddings as initial tokens’ representations and train or finetune a powerful decoder model to decode the whole sequences directly. We conduct extensive experiments to demonstrate that our generative inversion attack outperforms previous embedding inversion attacks in classification metrics and generates coherent and contextually similar sentences as the original inputs.

heading: 1 Introduction
text: Sentence embeddings serve as “universal embeddings” that have been widely used for numerous natural language processing tasks, e.g., text classification, question-answering, semantic retrieval, and other semantic similarity tasks (Cer et al., 2017). Recently, embedding models exploit large pre-trained language models to achieve revolutionary performance (Reimers and Gurevych, 2019; Gao et al., 2021). The notable improvement allows sentence embeddings to be directly used as inputs for downstream tasks. However, when applying sentence embeddings in downstream tasks, unintended data disclosure may violate the legislation, result in fines and hinder individuals from contributing their data to service models. For example, in some cases, such as legal document search,
when submitting a query embedding to a service based on neural models, we may not want to leak the sensitive information in the query embedding.
Language models have already been proven to memorize training data and some private training data can be extracted (Carlini et al., 2019, 2021; Thakkar et al., 2021). Besides the memorization issue, representations learned from language models also inherently leak sensitive information and suffer from attribute inference attacks (Song et al., 2017). Notably, for sentence embeddings, some of the words in the original sentence can be recovered, which is called embedding inversion attacks (Song and Raghunathan, 2020; Pan et al., 2020). As shown in Figure 1, both attribute inference attacks and embedding inversion attacks take sentence representations from language models as inputs. For attribute inference attacks, the adversary builds a multi-layer perceptron (MLP) to infer the input sentences’ private attributes (e.g., gender, race, and other identifiable personal information). For embedding inversion attacks, existing approaches are viewed as classification tasks that aim to recover partial keywords or unordered sets of original words from input sequences.
However, attribute inference attacks and existing embedding inversion attacks are not enough to explore the information leakage of sentence embeddings. If the malicious adversary can recover original sentences from their embeddings, attribute inference attacks and previous embedding inversion attacks can be conducted subsequently. Attribute inference attacks can be performed over the recovered sentences, which gives more flexibility for attack classifiers. Moreover, existing embedding inversion attacks can be done by directly converting the recovered sentences to sets (bag-of-words). Lastly, recovering the input sentences reveals more semantic information beyond the two attacks.
In this paper, we propose a generative embedding inversion attack (GEIA) to reconstruct input
14022
sentences given their embeddings from various language models. Unlike previous embedding inversion attacks, our attack is able to generate ordered sequences that share high contextual similarities with actual input sentences. Our proposed attack advances preceding embedding inversion attacks from classification to generation. Moreover, the generated sequences are mostly coherent and some are even verbatim text sequences from inputs. Finally, our attack is adaptive and effective to various LM-based sentence embedding models regardless of the models’ architectures and training methods. We perform extensive experiments to demonstrate the effectiveness of our attacks on Sentence-BERT (Reimers and Gurevych, 2019), SimCSE-BERT/SimCSE-RoBERTa (Gao et al., 2021), Sentence-T5 (Ni et al., 2022) and MPNet (Song et al., 2020). We also conduct experiments to show that our GEIA can even outperform previous attacks on classification metrics. Our contributions can be summarized as follows: 1
(1) To the best of our knowledge, we are the first to treat the embedding inversion attack as a generation task rather than a classification task. This allows our attack to reconstruct ordered sequences.
(2) Our GEIA can be adaptive to various embedding models with different model architectures from BERT to T5 and training algorithms like contrastive learning and siamese networks.
(3) We conduct extensive experiments to show the effectiveness of GEIA. Our results suggest that current state-of-the-art embedding models are vulnerable to GEIA.

heading: 2 Related Works
text: Sentence embedding with language models. Sentence embeddings aim to train universal vector rep-
1Code is publicly available at https://github.com/ HKUST-KnowComp/GEIA.
resentations that can handle downstream tasks. Earlier works learn sentence representations by exploiting encoder-decoder architectures to predict surrounding sentences (Kiros et al., 2015; Gan et al., 2017) and autoencoder models to reconstruct original sentences (Hill et al., 2016; Zhang et al., 2018a). Recent works turn to deeper and more complex transformer-based neural architectures like BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019) to further improve sentence representations. Sentence-BERT (Reimers and Gurevych, 2019) proposed a siamese network to improve the efficiency and performance of BERT representations. SimCSE (Gao et al., 2021) applied contrastive learning on BERT by self-predicting with dropout. Currently, Sentence-T5 (Ni et al., 2022) exploits T5 (Raffel et al., 2020) and contrastive learning to further improve embeddings on various tasks of semantic textual similarity (STS) (Conneau and Kiela, 2018).
Privacy leakage on language models. Even though language models bring dramatic improvements to sentence representations, there are rising privacy concerns on language models. Carlini et al. (2021) showed that language models tended to memorize training data and performed training data extraction attacks to recover private training data. Gupta et al. (2022) studied deep gradient leakage (Zhu et al., 2019; Zhao et al., 2020) in language models and extracted training texts from aggregated gradients. Besides the memorization issue, sentence embeddings from language models also encode private information that can easily be inferred by the adversary. Pan et al. (2020) recovered partial fixed patterns and keywords from language models’ sentence representations by querying language models with external annotated datasets. Similarly, Song and Raghunathan (2020) performed attribute inference attacks
and embedding inversion attacks to predict unordered sets of words from sentence embeddings. Though their attack setups are similar to our work, they simply view embedding inversion attacks as classification problems and cannot reconstruct input sequences. In this work, we reformulate the embedding inversion attack as a generation task and aim to invert ordered and informative sentences. Moreover, our proposed attack outperforms previous attacks even on classification metrics.

heading: 3 Embedding Inversion Attacks on LMs

heading: 3.1 Motivation
text: To show how embedding inversion attacks compromise privacy, we hereby define the breach of privacy first. Unintended or unauthorized data disclosure is regarded as privacy leakage during intended uses of the system. In our scenario, the intended uses refer to obtaining sentence embeddings from target embedding models while recovering original sentences through external attacker models (other than intended decoders of autoencoders) is unintended and unauthorized. When we use sentence embeddings for downstream applications and do not want to leak the original sentences, for example, personalized search in a neuralized document search engine in sensitive domains such as legal, medical, and financial domains, sensitive sentences may be unauthorizedly disclosed by curious service providers. Such unintended data disclosure may violate the legislation, incur fines and depress the service users.

heading: 3.2 Problem Formulation
text: Given a sensitive input text sequence x and a pretrained language model f on sentence representation, embedding inversion attacks aim to reconstruct the input x from its sentence embedding f(x). More specifically, the victim embedding model f is already pre-trained and its parameters are frozen. The adversary cannot update or modify the victim model’s architecture and parameters. Instead, the adversary holds an auxiliary dataset Daux that has a similar distribution to the data during attacks and attempts to build an external attacker model Φ to learn the inverse mapping f−1 such that:
Φ(f(x)) ≈ f−1(f(x)) = x. (1)
Due to the fact that sentence embeddings are commonly aggregated by pooling operations on final hidden representations of individual tokens, the mapping f that maps x to f(x) is inherently not one-to-one (injective). Thus, it is impossible for attacker model Φ to behave like the inverse mapping f−1. And it remains challenging for Φ to approximate f−1.

heading: 3.3 Existing Embedding Inversion Attacks
text: Previously, Song and Raghunathan (2020) considered both white-box and black-box attacks to recover sets of words from short input text sequences. For the white-box embedding inversion attack, it is assumed that the adversary can access the embedding model f ’s parameters and architecture. To make full use of the free access to f ’s parameters, the adversary first builds a model M that maps the
deep layer representation f(x) to its shallow layer representation. Then the recovered set of words x̂ are inferred from the shallow layer representation via continuous relaxation (Jang et al., 2017). For the black-box embedding inversion attack, the adversary can only query embedding model f with x from auxiliary data Daux to obtain its embedding f(x). Then the adversary directly learns Φ from (f(x), x) pairs through multi-label classification with MLP or multi-set prediction with RNN.

heading: 3.4 Limitations of Previous Approaches
text: Though these aforementioned embedding inversion attacks may recover some words from corresponding embeddings, existing approaches have several limitations. First, in the later experiment section, we show that existing embedding inversion attacks (multi-label classification and multi-set prediction) mainly predict insensitive stop words. Such attacks are incapable of inverting informative contents from sentences’ embeddings and therefore existing attacks are ineffective. Second, predicting sets of words cannot handle word repetitions well in a text sequence. Taking the epistrophe from the Bible as one example: “When I was a child, I spoke as a child, I understood as a child...” Simply predicting the token “child” can never capture the affluent linguistic properties. Lastly, such predicted sets are also orderless and semantic information of ordering is permanently lost. Taking the sentence “Alice likes Bob” as one example, even though we obtain the exact set of words: {Alice, likes, Bob}, we may still get the wrong meaning “Bob likes Alice.” As a result, existing approaches for recovering a set of words in a given sentence embedding are not as vicious as they claim.

heading: 3.5 Generative Embedding Inversion Attacks
text: To overcome the above limitations, we propose GEIA which attempts to generate sentences that are contextually similar to actual inputs. We follow the black-box setup that the adversary can only query the victim language model whose architecture and parameters are inaccessible. Intuitively, high-quality sentence embeddings encode rich linguistic properties about these sentences and a powerful generative decoder may utilize sentence embeddings to reconstruct original sentences. To invert a sequence given its embedding, we propose using a generative attacker model to decode words based on the embedding and previous contexts. As illustrated in Figure 2, the attacker model Φ can
exploit powerful language models like GPT-2 (Radford et al., 2019) to generate a sequence word by word from any given sentence embedding.
To train the attacker model, language modeling is applied with teacher forcing (Williams and Zipser, 1989) to generate a text sequence word by word:
LΦ(x; θΦ) = − u∑
i=1
log(Pr(wi|f(x), w0, w1, ..., wi−1)),
(2)
where x =“w0w1...wu−1” is a sentence of length u from the auxiliary data Daux and f(x) is the sentence embedding of x.
Unlike conventional encoder-decoder embedding models that intentionally and jointly train a decoder to strengthen the encoder, our GEIA solely trains the decoder based on the pre-trained and frozen embedding model f . By contrast, we treat the sentence embedding f(x) as the initial token representation before feeding the first word token w0 of x. Here, the token representation means the input to the first transformer block. If there is a size mismatch between the sentence embedding f(x) and the attacker model’s token representation, we apply one fully connected layer to align f(x) to be the same size as the attacker model’s token representation. We use Align(f(x)) to denote the aligned sentence embedding and Φemb(wi) to denote the representation of token wi. We concatenate Align(f(x)) to the left of the tokens’ representation to obtain the attacker’s input of x: [Align(f(x)),Φemb(w0),Φemb(w1), ...,Φemb(wu−1)] . This input bypasses Φ’s embedding layer and is directly fed to Φ’s first transformer block for text generation. As illustrated in Equation 2, the attacker Φ manages to maximize the probability of the target sequence [w0, w1, ..., wu−1,<eos>] given the input by minimizing the cross-entropy loss at each time step, where the <eos> indicates the special end of sentence token.
For inference, the attacker Φ decodes the first token from the aligned sentence embedding. Then tokens are generated iteratively from previous contexts with the sentence embedding till <eos> is reached. We use Φ(f(x)) to denote the whole generated sequence.

heading: 4 Experiments

heading: 4.1 Experimental Settings
text: Datasets. Most sentence embedding models are trained on question-answer pairs (semi-supervised
tasks) and natural language inference (supervised tasks) datasets. For our experiments, we evaluate the attacking performance on 2 datasets. The first dataset is PersonaChat (PC) dataset (Zhang et al., 2018b) that collects the open-domain chit-chat between two speakers given assigned personas. Most personas are reflected in corresponding utterances, and some of them can be sensitive and private. The second dataset is QNLI (Wang et al., 2018) converted from Stanford Question Answering Dataset (Rajpurkar et al., 2016). The QNLI dataset is collected from Wikipedia articles that include domain knowledge. Such domain knowledge can be challenging for inversion attacks. For evaluation, we use their training data as the auxiliary dataset to train the attacker model and report their testing performance, respectively. A summary of two datasets is shown in Table 2.
Sentence Embedding Models. To perform embedding inversion attacks, we consider the following 5 victim sentence embedding models: Sentence-BERT (Reimers and Gurevych, 2019), SimCSE-BERT/SimCSE-RoBERTa (Gao et al., 2021), Sentence-T5 (Ni et al., 2022) and MPNet (Song et al., 2020). All the embedding models’ parameters are frozen and the pre-trained weights in their original GitHub repository are used. Details and checkpoints of victim models are reported
in Appendix B. Attacker Models of Embedding Inversion. To compare with previous embedding inversion attacks, we implement two baseline attacker models proposed by Song and Raghunathan (2020).
• Multi-label classification (MLC). Given the embedding f(x) of sentence x. The adversary uses a simple MLP with binary cross entropy loss to predict words of x over the whole vocabulary.
• Multi-set prediction (MSP). MLC independently predicts words of x and ignores the dependency between words of a sequence. Multi-set prediction utilizes RNN architecture with multi-set prediction loss. We use the same multi-set objective that maximizes the probability of the set of tokens not predicted at the current time step as the previous work (Welleck et al., 2018).
• GEIA. As shown in Figure 2, our inversion attack can be regarded as sequence generation instead of set prediction. We train a GPT-2 as the attacker model from random weights. The random initialization makes a fair comparison between GEIA and previous baselines since both baselines also train from scratch. During our experiments, beam search decoding is applied for sentence recovery of GEIA. We also experiment on several attacker models with pre-trained weights and decoding methods. We put all the detailed evaluation results in the Appendix.
Evaluation Metrics. To make a fair comparison, our evaluation considers both classification and generation metrics.
The classification metrics include token-level micro precision/recall/F1 for previous embedding inversion and our generative inversion attacks. In addition, to study whether the recovered tokens are informative or not, we treat named entities as sensi-
tive information and stop words as non-informative tokens. We use the named entity recovery ratio (NERR) to measure the percentage of name entities that can be retrieved from input texts. We also propose the stop word ratio (SWR) to calculate the percentage of stop words for given sentences. Vicious embedding inversion attacks can achieve high NERRs with similar SWRs of original sentences.
Different from previous works, our GEIA can generate sequences instead sets of words. So we also evaluate GEIA on the sentence-level generation metrics. We apply ROUGE (Lin, 2004), BLEU (Papineni et al., 2002), and embedding similarity (ES) to measure the similarity between input sentences and inverted sequences. ROUGE and BLEU both measure similarity based on n-grams. ROUGE focuses on recall: how much the n-grams in the inputs are recovered. BLEU measures precision: how much the n-grams inverted are correct. The embedding similarity exploits “sentence-t5xxl” to compute cosine similarity for the semantic similarity. In addition, we use perplexity (PPL) of fine-tuned “gpt2-large” to measure the fluency of generated sentences.

heading: 4.2 Evaluation on Classification
text: Firstly, we compare the attacking performance between previous inversion baselines (MLC and MSP) and our GEIA on classification metrics.
Table 1 shows the attacking performance of previous inversion attacks and our GEIA on the testing data. We report token-level micro precision (Pre), recall (Rec), and F1 for all 3 attacks. For multilabel classification, we carefully tune the binary thresholds based on the validation set and report the thresholds with the highest F1 scores. By comparison, it can be seen that our GEIA outperforms MLC and MSP. Except MSP has a slightly higher
F1 on SimCSE-RoBERTa of the QNLI dataset, our GEIA has the highest recall and F1 on 5 victim embedding models for both datasets. On the PersonaChat dataset, our GEIA has the dominating performance with F1 scores of around 0.58 while MSP has F1 scores of around 0.4 and MLC has an average F1 score of no more than 0.3. These results show that our generative inversion attack has better attacking performance on classification tasks even though it is designed for generating sequences.
Despite tenable classification performance, previous MLC and MSP have several limitations. For MLC, we find that most predicted tokens’ probabilities are close to the reported thresholds and it is hard to distinguish tokens on MLC. For MSP, though it performs better than MLC, it cannot handle long time steps well. And the high precision on the QNLI dataset also benefits from small time steps. In addition, they both tend to invert uninformative stop words and most inverted results include no sensitive content.The subsequent experiments can help verify these stated limitations.

heading: 4.3 Precision-Recall Trade-off of MLC
text: To better understand limitations of the baseline classifier’s performance on embedding inversion, we draw the precision-recall curve for MLC with different thresholds. Figure 3 shows the attack results for all victim embedding models where the individual result is marked by the marker. Most data points are clustered in the upper-left zone of the figure. The result indicates that MLC frequently leads to high precision with low recall. As the threshold increases, for attacking performance of all five models, the precision drops dramatically compared with a minor increase in the recall.
Regardless of the high precision, most areas under the curves are small. The small areas imply that MLC is not a good classification model. High precision means that MLC makes most predictions correctly while low recall means that MLC returns very few predictions. Since the distribution among tokens is highly imbalanced and NNs may easily overfit the training data, it remains unknown whether the predicted tokens are sensitive or not.

heading: 4.4 What Types of Tokens are Inverted?
text: Though the classification performance is evaluated, the informativeness of inverted tokens remains unknown. Here we study the sensitivity of recovered tokens based on the named entity recovery ratio (NERR) and stop word rate (SWR). A menac-
ing embedding inversion attack can recover most named entities of original sequences. If the recovered tokens are mostly stop words, the embedding inversion should be regarded as a failure.
Table 3 includes NERRs and SWRs for MLC, MSP and GEIA. For NERR, both MLC and MSP can only recover less than 3% named entities from sentence embeddings for all situations. On average, our GEIA can invert around 40% named entities on the PersonaChat dataset and around 15% named entities on the QNLI dataset. The results on NERRs suggest that our GEIA can indeed recover sensitive content while previous baselines fail to capture informative entities from embedding inversion attacks. For SWR, we first report the SWRs of the original datasets and then calculate the SWR differences between inverted results and corresponding input sentences. Our generative inversion attacks have the slightest difference in most cases. We investigate MLC’s results of SimCSE-BERT and SimCSE-RoBERTa on both datasets to see why their SWRs differ from other models. For SimCSE-BERT on MLC, their SWRs
are much smaller than the corresponding datasets’ SWRs. This is caused by several extreme cases that invert more than 10,000 tokens on embeddings of SimCSE-BERT and hence SWRs of MLC are around -20%. For SimCSE-RoBERTa on MLC, we find a small number of cases that output hundreds of irrelevant tokens. Hence their SWRs are close to the datasets’ SWRs by chance.
Results from both SWR and NERR confirm that previous embedding inversion attacks have little threat of breaching privacy while our GEIA can indeed recover sensitive information.

heading: 4.5 Evaluation on Generation
text: Besides the classification performance and inverted tokens’ sensitivity, the generation quality is also vital to generative embedding inversion attacks. Here we view the inverted sequences as translated sentences and use generation metrics from the machine translation task to compare the similarity between inverted sequences and original inputs. In addition, we also tune another GPT-2 model (pre-trained GPT-2medium) with training data as the baseline.
During inference, this GPT-2 tries to decode a sentence given the first token with/without context. Here, the context refers to all previous contents of the sentence. For conversions of the PersonaChat dataset, the context means all the previous utterances of the current utterance in a conversation. Our GEIA tries to invert the sentence given the embedding of this sentence.
We perform a generation evaluation on the PersonaChat dataset and the results are shown in Table 4. We can view BLEU-1 and ROUGE-1 as word-level precision and recall of inverted sequences, respectively. For all 5 victim models, ROUGE-1 ranges from 0.59 to 0.72 and BLEU-1 ranges from 0.35 to 0.46. That is, more than half of the words in the original inputs are recovered and one-third of inverted words are also in the inputs. The results on ROUGE-1 and BLEU-1 are consistent with token-level precision and recall in Table 1. By comparing ROUGE and BLEU with baselines, our GEIA is much more effective for inverting ngrams. What is more, the embedding similarities of our generative inversion attacks are much higher than the GPT-2 baseline with the given context. For fluency, we also compare the perplexity between our attacks and baselines. Since the GPT-2 is tuned and the length of each sentence is relatively short, perplexities are small for the results. Still, our GEIA can generate sequences with lower PPL than two baselines. These results verify that our generative embedding attack can generate sequences well with inverted n-grams. A more detailed evaluation of the generation is shown in the Appendix.

heading: 4.6 Case Study
text: To show how our generative inversion attack outperforms previous baselines, we give an example of inverted results on different victim models in Fig-
ure 4. We manually highlight informative words in the input sentence. If the inverted results have the same meaning as the informative words, we also highlight them using corresponding colors. The figure shows that almost no informative word is inverted for the two baselines and the inverted tokens are mostly stop words and punctuation marks. Therefore, it is impractical to infer the semantics for existing inversion attacks. For our GEIA, all the inverted results can recover “love plants” and 3 out of 5 results can invert “hiking” and “environmental activist”. Thus, the semantics of the input sentence can be easily captured. Despite admissible precision and recall from previous embedding inversion attacks, this example exemplifies that previous attacks cannot recover meaningful words. On the other hand, these results justify that our generative embedding inversion attack can indeed invert informative words from sentence embeddings. We leave more detailed contents and analyses of case studies in Appendix G.

heading: 5 Conclusion
text: In this paper, we show the limitations of existing inversion attacks and propose a generative embedding inversion attack to better recover original text sequences given their sentence embeddings. Then we show that LM-based sentence embedding models are potentially vulnerable to our proposed attack. We conduct extensive experiments to demonstrate the inability of previous embedding inversion attacks and disclose the privacy risks from our attack. The defenses against the embedding inversion attack are not well studied yet, even though it is much more malicious than the attribute inference attack. For future work, we call for more attention to effective defenses to address the embedding inversion attack with minor costs.
Limitations
From the adversary’s perspective, our attacker model’s main limitation is the incapability of recovering exact domain-specific tokens. During our experiments, we evaluate attacking results on the PersonaChat and QNLI datasets. The PersonaChat dataset collects daily conversations between speakers with almost no expert knowledge. The QNLI includes question-answer pairs from Wikipedia with far more domain-specific named entities than the PersonaChat dataset. By comparing the attacking evaluations in Table 1, 3 and 10, all attacks on the PersonaChat dataset are more successful than attacks on the QNLI dataset. For instance, in Table 1, F1 scores on PC are 0.1∼0.2 larger than on QNLI on average. In addition, QNLI 2 of Figure 5 shows that GEIA fails to recover the exact location “Fresno” 7 out of 10 times. Though most inverted results are similar to “What was the population of the city in 2010?” It is hard to capture the exact city name “Fresno”.
Ethical Considerations
We declare that all authors of this paper acknowledge the ACM Code of Ethics and honor the code of conduct. This work substantially reveals potential privacy vulnerabilities of current LM-based sentence embedding models during inference. We hereby propose the generative embedding inversion to further exploit the weaknesses of those widely used sentence embedding models. We hope to raise more awareness of privacy leakage inside sentence embeddings and call for defenses against such information leakage.
Data. During our experiment, no personal identifiable information is used or revealed. Both PersonaChat and QNLI are publicly available datasets and anonymity is applied during data collection.
Victim Embedding Models. For our experiment, we use the sentence embedding models from the original GitHub repositories with given weights. In the future, if there are other open-sourced embedding models with improved protection on privacy, we will test our proposed attack on them.

heading: Acknowledgment
text: The authors of this paper were supported by the NSFC Fund (U20B2053) from the NSFC of China, the RIF (R6020-19 and R6021-20) and the GRF (16211520 and 16205322) from RGC of Hong
Kong, the MHKJFS (MHP/001/19) from ITC of Hong Kong and the National Key R&D Program of China (2019YFE0198200) with special thanks to HKMAAC and CUSBLT. We also thank the UGC Research Matching Grants (RMGS20EG01-D, RMGS20CR11, RMGS20CR12, RMGS20EG19, RMGS20EG21, RMGS23CR05, RMGS23EG08).

heading: A Training Details
text: Hyper-parameters and setups. For the multilabel classification, we use a 1-layer neural network as the attacker. To determine the thresholds for classification, we perform the grid search with an interval of 0.05. The multi-set prediction uses a unidirectional GRU (Chung et al., 2014) of 10 time steps as the attacker. For every time step, we use sentence embedding as the input with the multi-set objective (Welleck et al., 2018). We experiment with different time steps and inputs (e.g., averaging the sentence embedding with corresponding token embedding for a time step) and find that the time step of 10 with only sentence embeddings yields the best performance. The baselines and our GEIA use the same byte pair encoding tokenizer (Sennrich et al., 2016) for a fair comparison. Our GEIA uses a randomly initialized GPT-2 medium model (345M) (Radford et al., 2019). The decoding uses the beam search with beam size 5. For all 3 models, we use the Adam optimizer to update the models with a learning rate of 3e-4 and batch size of 64.
Training details. During training, we first obtain batches of sentence embeddings from victim embedding models and project embeddings to the exact dimension of the attacker’s token representations. Then we gather the corresponding batches of tokens’ representations by passing tokens through the attacker’s embedding layers with attention masks. As shown in Figure 2, we concatenate sentence embeddings to the left of the tokens’ representations as the inputs. Hence, the sentence embeddings can be viewed as the initial token representations followed by the original tokens’ representations. Lastly, we feed concatenated representations to train the attacker with the language modeling objective.
Take the sentence x =“w0w1...wu−1” as one example, we use Align(f(x)) to denote the aligned sentence embedding and Φemb(wi) to denote the representation of token wi of attacker model Φ. Our input I is [Align(f(x)),Φemb(w0),Φemb(w1), ...,Φemb(wu−1)] . And our prediction manages to maximize the probability of the target sequence O = [w0, w1, ..., wu−1,<eos>], where the <eos> indicates the special end of sentence token. Both I and O are of length u+ 1. For each time step t where 0 < t < u, our attacker aims to output Ot = wt given Align(f(x)), Φemb(w0), ...,Φemb(wt−1). If t = 0, the desired output is O0 = w0 given only
the sentence embedding Align(f(x)). If t = u, the desired output is Ou = <eos> given the whole input sequence I .
Toolkits. For finding named entities, we use Stanza toolkit (Qi et al., 2020) to extract named entities from two datasets. We use the NLTK package to measure BLEU scores and Huggingface’s Evaluate library to measure ROUGE scores. For micro-averaged scores, we use the sklearn library to calculate precision, recall and F1.
During our experiment, we use 2 NVIDIA GeForce RTX 3090 to run our codes and it takes around 7 hours to train the attacker of GEIA for 10 epochs.

heading: B Details of Victim Embedding Models
text: In this section, we give more details of victim embedding models used in our experiments and their checkpoints.
• Sentence-RoBERTa (SRoBERTa) (Reimers and Gurevych, 2019): Sentence-BERT proposes a siamese network to reduce computational overhead for sentence embeddings. We adopt the SentenceRoBERTa model since it has better performance than SBERT. In our experiment, “all-roberta-largev1” (355M) is used as SRoBERTa.
• SimCSE (Gao et al., 2021): SimCSE considers the simple contrastive learning objective by self-prediction with dropout. And SimCSE performs better than SBERT and SRoBERTa. For our experiment, we use both SimCSE-BERT (“supsimcse-bert-large-uncased,” 340M) and SimCSERoBERTa (“sup-simcse-roberta-large,” 355M) as victim models.
• Sentence-T5 (ST5) (Ni et al., 2022): SentenceT5 exploits the encoder of T5 model architecture (Raffel et al., 2020) to achieve the new state-of-theart on sentence embedding tasks. In our experiment, “sentence-t5-large” (770M) is used.
• MPNet (Song et al., 2020): MPNet proposed a unified learning objective for BERT to combine masked language modeling and permuted language modeling. In our experiment, “all-mpnet-base-v1” (110M) is used.

heading: C Evaluations on Exact Match Ratio and Edit Distance
text: In addition to the similarity, we are also interested in how many inverted sequences are verbatim input sentences. If they are not the same, we would like to know the minimal edits to modify the inverted
sequence to the inputs. In this part, we use the exact match ratio (EMR) to calculate the ratio of inverted sequences that are exactly the same as inputs after removing punctuation. We also report the mean and median of edit distance (ED). The edit distance, also known as Levenshtein distance, measures the minimal changes of characters needed to update the inverted sequence to the actual input.
The evaluation results are shown in Table 10. For EMR, our GEIA can recover approximately 10% of verbatim sentences on the PersonaChat dataset. However, our GEIA inverts no more than 1% exact sequences on the QNLI dataset. For edit distance, GEIA needs around 28 edits on the PersonaChat dataset and 85 edits on the QNLI dataset. These results show that our GEIA can indeed recover verbatim input sentences from their embeddings. Still, GEIA cannot handle domain knowledge well and the performance drops on the QNLI dataset.

heading: D Evaluation on Decoding Methods
text: Our experiments implement beam search decoding for the generation process. In this section, we compare beam search decoding with sampling-based decoding. We use the Nucleus Sampling (Holtzman et al., 2020) method to sample tokens. We set top-p = 0.9 with a temperature coefficient 0.9.
The evaluation results of the two decoding meth-
ods are shown in Table 10. Both generation and classification metrics are included in the two datasets. Except for a few results of BLEU-1 and Recall on the QNLI dataset, beam search significantly outperforms Nucleus Sampling on various metrics. For example, compared with Nucleus Sampling, beam search brings 3% - 5% improvements on the F1 and 1% - 3% improvements on the BLEU4. Additionally, beam search leads to higher EMRs and smaller edits. These results help explain why we use the beam search decoding for our experiments.

heading: E Evaluations on Models’ Sizes
text: In this section, to study the attack performance on LM-based embedding models of different scales, we perform generative embedding inversion attacks on a specific victim model with different model sizes and attacker sizes.
In Table 5, we evaluate GEIA on pre-trained ST5 of four different model sizes from ST5base (220M) to ST5xxl (11B) on the PersonaChat dataset. We use GPT-2medium (345M) and GPT-2large (762M) as attackers. The good attacking results suggest that GEIA is still effective despite different model scales. By comparing attacking performance on the same victim model between GPT-2medium and GPT-2large, we found that embedding models are generally more vulnerable after increasing the at-
tacker’s capacity. In addition, we find that ST5base tends to be the most vulnerable to GEIA while other models are more robust towards GEIA. This suggests that small-sized embedding models are more unsafe towards GEIA than the larger models.

heading: F Evaluations on Different Attacker Models and Initializations
text: To show that GEIA can be adaptive with various powerful decoders, we evaluate the performance of GEIA with different attacker models and initializations on the PersonaChat dataset with ST5large (770M) as the victim model. Besides GPT-2, we extend attacker models to OPT (Zhang et al., 2022) and T5 and perform GEIA on ST5large. GPT-2 and OPT are built on the transformer’s decoder blocks and pre-trained with different datasets while T5 consists of both encoder and decoder blocks. For GEIA with T5 as the attacker, we feed sentence embeddings to the encoder and perform decoding on the decoder side. For GPT-2, we use randomly initialized GPT-2medium-random (345M), pre-trained GPT-2medium (345M) and pre-trained GPT-2large (762M) as attackers. For OPT, we experiment with pre-trained OPT350m and OPT1.3b. For T5, we evaluate performance on randomly initialized T5large-random (770M) and LM-adapted T5large-lm-adapt2 (770M).
In Table 6, we can see that all our attackers can outperform the previous baselines of Table 1 on classification. Interestingly, our results suggest that pre-training does not constantly improve performance than random initialization for GEIA. In terms of beam search, Pretrained T5large-lm-adapt outperforms T5large-random while GPT-2medium-random defeats both GPT-2medium and GPT-2large. Since our GEIA has a different training paradigm from LMs’ pre-training objectives, it is hard to conclude whether pre-training can improve GEIA or not. However, pre-training helps attackers better model the probability distribution of tokens and enhances the performance of sampling-based decoding strategies. In addition, our results on model scales of GPT-2 and OPT help verify that GEIA can be improved by increasing attackers’ sizes.
2https://huggingface.co/google/ t5-large-lm-adapt.

heading: G More on Case Studies
text: In this section, we give more cases on two datasets to show the effectiveness of GEIA intuitively. Figure 5 gives two examples for each dataset with 2 decoding methods included. Still, the informative words are highlighted manually for both input sentences and inverted results. For all cases, MLC performs the worst: only the token “love” is inverted 3 times on the first example of PC. And the remained inverted results are mostly meaningless stop words. MSP performs better than MLC: some informative words like “US” and “population” can be recovered on the QNLI dataset. For GEIA, both decoding algorithms can recover many relevant words and generate coherent sentences. Moreover, some digits can even be mined: the year “2010” of QNLI 2 is successfully recovered for 9 out of 10 cases. On QNLI 1, the number “50” is also inverted by GEIA on ST5. Interestingly, both GEIA also try to predict the numbers during generation: beam search decoding predicts “45” and “51” while Nucleus Sampling outputs “51.” This example suggests that GEIA is also aware of digits like years. On PC 2, GEIA can even capture all 3 hobbies and invert them correctly.
In summary, these cases show that previous MLC and MSP perform poorly on embedding inversion and our GEIA works much better than previous works.

heading: H GEIA on More Datasets
text: To demonstrate that our proposed GEIA is universal and can be applied to any textual data, here we evaluate GEIA on 5 more datasets of different domains, tasks and scales. Without loss of generality, we set SimCSE-BERT as the victim model and perform inversion attacks on Action-Based Conversations Dataset (ABCD) (Chen et al., 2021), Multi-Genre Natural Language Inference (MNLI) (Williams et al., 2018), Multi-domain Wizard-ofOz (MultiWOZ) (Budzianowski et al., 2018), Stanford Sentiment Treebank v2 (SST2) (Socher et al., 2013) and WMT16 (Bojar et al., 2016). A detailed summary statistics of these five datasets is shown in Table 7.
Table 8 and 9 compares GEIA with previous baselines in classification and informativeness, separately. We can still observe that our GEIA can mostly outperform MLC and MSP with better recall and F1 on both classification and informativeness. One exception is that MSP outperforms GEIA
on SST-2. We examined the inverted contents of GIEA on SST-2, and found that GIEA frequently generated repetitions of meaningless stop words due to insufficient training data (merely 67,349 sentences). The randomly initialized GPT-2 cannot generalize well on this small-scale dataset and therefore performs poorly on embedding inversion.
After inspecting the attacking performance on all seven datasets, including PC and QNLI, we discovered that embedding inversion attacks’ performance is dependent on data scales, domains and informativeness of contents. Still, given a reasonable amount of training data, our GEIA can easily exceed previous baselines on both classification and informativeness. Moreover, our proposed GEIA changes the previous classification paradigm to generation and can recover ordered sequences.
D at
a V
ic tim
M od
el In
iti al
iz at
io n
PP L
E S
R O
U G
E B
L E
U C
la ss
ifi ca
tio n
E M
R E
D
D ec
od e
R O
U G
E -1
R O
U G
E -L
B L
E U
-1 B
L E
U -2
B L
E U
-4 Pr
e R
ec F1
M ea
n M
ed ia
n
PC
SR oB
E R
Ta 7.
52 83
.7 7
44 .5
5 41
.0 0
30 .1
6 13
.3 1
9. 75
39 .6
0 39
.5 4
39 .5
7 8.
71 33
.1 0 35 Si m C SE -B E R T G PT -2 pr et ra in 8. 70 86 .5 6 54 .8 3 48 .9 4 38 .2 4 19 .0 1 14 .1 5 46 .8 1 47 .2 4 47 .0 3 11 .0 5 31 .1 6 32 Si m C SE -R oB E R Ta — — — — — — 8. 46 86 .4 8 50 .5 6 44 .9 0 35 .2 0 16 .0 4 11 .5 3 44 .0 9 44 .1 3 44 .1 1 8. 43 32 .6 3 34 ST 5 N uc le us 8. 45 88 .3 1 55 .5 5 50 .2 1 38 .4 2 19 .1 8 14 .2 9 48 .4 2 48 .3 2 48 .3 7 11 .5 4 29 .8 3 31 M PN et 8. 30 85 .8 8 50 .4 9 45 .8 8 34 .5 3 16 .2 1 12 .0 3 44 .3 2 44 .3 1 44 .3 2 10 .4 4 31 .4 5 33 SR oB E R Ta 4. 78 85 .8 1 50 .6 8 47 .3 0 31 .6 2 16 .1 9 12 .2 2 47 .9 2 42 .6 5 45 .1 3 10 .4 0 29 .6 3 30 Si m C SE -B E R T G PT -2 pr et ra in 6. 68 88 .2 3 60 .0 2 54 .0 9 41 .0 2 22 .2 5 16 .8 5 52 .5 2 50 .8 8 51 .6 9 12 .9 9 28 .4 3 29 Si m C SE -R oB E R Ta — — — — — — 5. 33 88 .1 9 56 .4 5 51 .1 0 36 .2 2 18 .8 4 14 .0 1 51 .8 6 46 .7 8 49 .1 9 10 .4 3 28 .9 5 30 ST 5 B ea m 5. 50 90 .2 6 62 .1 8 57 .1 1 40 .1 0 22 .8 4 17 .5 3 57 .2 0 51 .6 9 54 .3 1 14 .4 8 26 .1 5 26 M PN et 5. 39 87 .8 9 56 .8 2 52 .3 8 36 .2 1 19 .5 6 14 .9 0 52 .7 8 47 .6 8 50 .1 0 12 .6 9 27 .8 7 28 SR oB E R Ta 9. 27 86 .5 3 55 .0 3 51 .0 5 36 .7 5 18 .6 5 13 .8 9 49 .5 8 47 .8 7 48 .7 1 11 .8 0 28 .5 3 29 Si m C SE -B E R T G PT -2 ra nd om 10 .1 4 89 .9 4 68 .2 0 60 .9 9 47 .2 3 26 .9 5 20 .8 1 59 .7 0 58 .2 3 58 .9 5 17 .2 4 25 .4 9 25 Si m C SE -R oB E R Ta — — — — — — 9. 71 90 .0 2 64 .6 2 57 .9 3 43 .8 6 23 .6 8 17 .8 7 57 .1 1 55 .3 4 56 .2 1 14 .2 9 26 .7 6 27 ST 5 N uc le us 10 .2 4 90 .1 6 66 .6 6 60 .8 1 45 .9 1 26 .2 4 20 .1 5 59 .2 0 57 .5 2 58 .3 5 16 .8 1 24 .7 9 24 M PN et 10 .3 2 87 .8 3 60 .8 4 55 .9 5 41 .3 7 22 .2 0 16 .7 6 54 .1 7 52 .7 5 53 .4 5 14 .7 2 26 .5 7 27 SR oB E R Ta 4. 99 88 .0 7 59 .5 4 56 .0 4 35 .4 7 20 .3 7 15 .6 6 58 .4 1 48 .9 1 53 .2 4 13 .4 1 26 .3 4 26 Si m C SE -B E R T G PT -2 ra nd om 6. 29 91 .2 8 72 .3 8 65 .3 3 46 .9 3 28 .9 9 22 .8 5 66 .9 5 59 .6 9 63 .1 1 18 .9 6 23 .1 8 22 Si m C SE -R oB E R Ta — — — — — — 5. 98 91 .3 3 68 .7 8 62 .4 2 43 .4 1 25 .6 6 19 .8 2 64 .2 7 56 .6 6 60 .2 2 16 .0 4 24 .3 0 23 ST 5 B ea m 5. 90 91 .4 7 70 .7 2 65 .4 5 44 .5 2 27 .8 3 21 .9 9 67 .4 6 58 .2 6 62 .5 3 19 .1 1 22 .6 8 21 M PN et 5. 64 89 .2 7 65 .0 8 60 .3 9 40 .0 4 23 .8 3 18 .5 4 62 .6 4 53 .5 1 57 .7 2 16 .6 3 24 .6 6 23
Q N
L I
SR oB
E R
Ta 25
.4 8
80 .1
3 26
.6 5
22 .4
5 20
.3 2
3. 95
2. 47
28 .4
0 27
.0 9
27 .7
3 0.
17 95
.9 9 75 Si m C SE -B E R T G PT -2 pr et ra in 26 .3 6 80 .4 5 30 .7 1 26 .2 2 23 .0 3 5. 21 3. 50 30 .2 7 29 .0 7 29 .6 6 0. 21 91 .2 1 70 Si m C SE -R oB E R Ta — — — — — — 22 .8 5 80 .1 3 27 .8 5 23 .7 6 20 .5 4 4. 26 2. 75 29 .1 2 27 .2 6 28 .1 6 0. 22 94 .9 7 74 ST 5 N uc le us 28 .7 4 83 .2 1 32 .7 7 27 .8 6 24 .1 1 5. 86 3. 91 31 .8 4 30 .6 9 31 .2 5 0. 37 90 .7 1 71 M PN et 28 .6 6 81 .4 6 29 .9 4 25 .1 1 22 .5 7 4. 88 3. 19 30 .3 8 29 .2 3 29 .8 0 0. 33 93 .7 2 74 SR oB E R Ta 11 .0 9 81 .1 8 30 .5 4 26 .2 3 20 .4 0 4. 97 3. 28 35 .0 2 27 .3 3 30 .7 0 0. 32 87 .4 7 67 Si m C SE -B E R T G PT -2 pr et ra in 11 .7 4 81 .3 7 34 .2 4 29 .8 3 22 .6 3 6. 24 4. 35 36 .8 4 28 .8 7 32 .3 7 0. 50 84 .7 7 65 Si m C SE -R oB E R Ta — — — — — — 10 .4 3 81 .3 4 31 .6 9 27 .4 6 20 .5 0 5. 27 3. 53 35 .6 9 27 .4 7 31 .0 4 0. 41 87 .1 3 67 ST 5 B ea m 12 .0 6 84 .1 5 36 .5 8 31 .7 5 23 .3 1 6. 82 4. 70 39 .0 5 30 .3 0 34 .1 3 0. 75 83 .7 9 64 M PN et 12 .5 9 82 .5 2 33 .8 8 28 .9 2 22 .2 1 5. 84 3. 94 37 .1 8 29 .2 3 32 .7 3 0. 58 85 .9 7 66 SR oB E R Ta 91 .7 1 80 .1 7 31 .7 5 26 .9 4 23 .9 8 5. 68 3. 65 32 .6 1 30 .3 4 31 .4 4 0. 36 89 .8 1 67 Si m C SE -B E R T G PT -2 ra nd om 94 .5 1 81 .7 9 37 .8 3 32 .3 1 27 .9 9 7. 73 5. 33 36 .1 5 33 .8 0 34 .9 4 0. 61 86 .0 1 63 Si m C SE -R oB E R Ta — — — — — — 78 .1 2 82 .5 6 36 .9 0 31 .3 1 26 .3 5 7. 20 4. 98 37 .0 6 33 .3 1 35 .0 9 0. 53 86 .9 9 64 ST 5 N uc le us 98 .8 7 81 .4 3 36 .0 5 31 .1 4 26 .7 3 7. 31 4. 90 34 .8 4 32 .6 0 33 .6 8 0. 41 85 .8 6 63 M PN et 97 .5 5 79 .8 1 33 .5 5 28 .5 2 25 .0 6 6. 17 4. 11 33 .2 4 31 .0 8 32 .1 2 0. 46 87 .8 5 66 SR oB E R Ta 11 .1 3 81 .0 4 34 .3 4 29 .9 7 18 .5 2 5. 48 3. 67 43 .8 1 27 .1 9 33 .5 6 0. 50 84 .7 9 61 Si m C SE -B E R T G PT -2 ra nd om 10 .8 3 82 .2 4 40 .0 1 35 .2 5 20 .5 8 7. 19 5. 08 48 .7 8 29 .4 9 36 .7 6 0. 85 83 .1 4 58 Si m C SE -R oB E R Ta — — — — — — 10 .6 9 83 .1 2 38 .7 3 33 .7 6 19 .4 7 6. 54 4. 63 48 .6 2 29 .2 6 36 .5 3 0. 81 83 .8 8 60 ST 5 B ea m 10 .8 1 82 .0 5 38 .1 9 33 .9 3 19 .5 3 6. 81 4. 77 47 .4 2 28 .4 3 35 .5 5 0. 52 83 .3 2 59 M PN et 11 .4 1 80 .7 4 36 .1 8 31 .6 5 19 .0 4 5. 98 4. 15 44 .8 9 27 .7 4 34 .2 9 0. 60 84 .1 7 60
Ta bl
e 10
: A
co m
pl et
e ev
al ua
tio n
of ge
ne ra
tio n
qu al
ity w
ith di
ff er
en td
ec od
in g
al go
rit hm
s an
d m
od el
in iti
al iz
at io
ns .E
M R
re fe
rs to
th e
ex ac
tm at
ch ra
tio an
d ED
st an
ds fo
rt he
ed it
di st
an ce
.E m
be dd
in g
Si m
ila ri
ty ,R
O U
G E
,B L
E U
,P re
,R ec
,F 1
an d
E M
R ar
e m
ea su
re d
in %
.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: A1. Did you describe the limitations of your work?
Left blank.
A2. Did you discuss any potential risks of your work? Left blank.
A3. Do the abstract and introduction summarize the paper’s main claims? Left blank.
A4. Have you used AI writing assistants when working on this paper? Left blank.

heading: B Did you use or create scientific artifacts?
text: Left blank.
B1. Did you cite the creators of artifacts you used? Left blank.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Left blank.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Left blank.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Left blank.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Left blank.

heading: C Did you run computational experiments?
text: Left blank.
C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Left blank.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Left blank.
C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Left blank.
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Left blank.

heading: D Did you use human annotators (e.g., crowdworkers) or research with human participants?
text: Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Left blank.

heading: Future Work
text: For future work, we call for more attention to effective defenses to address the embedding inversion attack with minor costs. Limitations
From the adversary’s perspective, our attacker model’s main limitation is the incapability of recovering exact domain-specific tokens. During our experiments, we evaluate attacking results on the PersonaChat and QNLI datasets. The PersonaChat dataset collects daily conversations between speakers with almost no expert knowledge. The QNLI includes question-answer pairs from Wikipedia with far more domain-specific named entities than the PersonaChat dataset. By comparing the attacking evaluations in Table 1, 3 and 10, all attacks on the PersonaChat dataset are more successful than attacks on the QNLI dataset. For instance, in Table 1, F1 scores on PC are 0. 1∼0. 2 larger than on QNLI on average. In addition, QNLI 2 of Figure 5 shows that GEIA fails to recover the exact location “Fresno” 7 out of 10 times. Though most inverted results are similar to “What was the population of the city in 2010?” It is hard to capture the exact city name “Fresno”