Detecting and Perturbing Privacy-Sensitive Neurons to Defend Embedding Inversion Attacks

Authors: Yuche Tsai, Hsiang Hsiao, Shou-De Lin
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: Text embedding, Defense Inversion Attack

Abstract:
This paper introduces Defense through Perturbing Privacy Neurons (DPPN), a novel approach to protect text embeddings against inversion attacks. Unlike ex- isting methods that add noise to all embedding dimensions for general protection, DPPN identifies and perturbs only a small portion of privacy-sensitive neurons. We present a differentiable neuron mask learning framework to detect these neu- rons and a neuron-suppressing perturbation function for targeted noise injection. Experiments across six datasets show DPPN achieves superior privacy-utility trade- offs. Compared to baseline methods, DPPN reduces more privacy leakage by 5-78% while improving downstream task performance by 14-40%. Tests on real- world sensitive datasets demonstrate DPPN’s effectiveness in mitigating sensitive information leakage to 17%, while baseline methods reduce it only to 43%.

Decision: reject
Reviewer scores: 6, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1729161792396

thread_type:
official_review

<thread_object>
creator: reviewer_ffat

<content>
summary:
The paper proposes a defense method DPPN to resist against embedding inversion attacks. In contrast to previous methods that add noise on all embedding dimensions, it recognize privacy neurons that contains more sensitive information and only perturbs them. Therefore, it achieve an excellent privacy-utility trade-off. Extensive experiments show that the defense method can protect private information while maintaining the origin semantic information.

strengths:
Originality: The introduction of privacy neurons and targeted perturbation is innovative, departing from conventional methods that apply noise to all dimensions.

Quality: The conducted experiments are comprehensive. The study evaluates DPPN across six datasets, multiple embedding models, and various attack models, showcasing robust and thorough experimental design.

Clarity: The paper follows a clear structure, with a logical flow from problem motivation to solution, experiments, and results.

Significance: The ability to reduce privacy leakage without sacrificing utility makes DPPN relevant for real-world applications where maintaining both privacy and accuracy is critical.

weaknesses:
+ The paper does not provide sufficient detail on how parameters $\xi$ and $\eta$ in formulas 3 and 5 are selected during the experiments. Please provide additional information about the selection process. This clarification would help readers understand their impact and ensure reproducibility.
+ The concepts of $D^+$ and $D^-$ are introduced on line 170, but their explanations are deferred until line 204. This gap may confuse readers. It would be clearer if a brief definition were provided when these concepts are first mentioned, or if the detailed explanation were moved closer to the first appearance.
+ In Fig. 2, the paper introduces the concept of sensitivity for privacy neurons. What if privacy neurons were selected based on their top dimension-wise sensitivity? The paper lacks a study on it.

questions:
+ The paper uses a fixed top-k selection method for privacy neurons. What if we choose them based on a threshold of $m$, such as 0.5?
</content>
</thread_object>

</thread>
<thread>
creation date:
1729670010208

thread_type:
official_review

<thread_object>
creator: reviewer_8zq5

<content>
summary:
The paper presents DPPN, a method for defending text embeddings against inversion attacks by selectively perturbing privacy-sensitive neurons. It demonstrates strengths in improving privacy-utility tradeoffs and shows robustness across different datasets and embedding models. However, it also has weaknesses, including the lack of theoretical privacy guarantees and potential challenges in generalizing to new data. Overall, DPPN is a promising approach for privacy protection in text embeddings, but further research is needed to address its limitations and ensure broader applicability.

strengths:
Strengths:

1. Targeted Defense: DPPN focuses on identifying and perturbing only a subset of privacy-sensitive neurons, which is a more efficient approach compared to perturbing all dimensions of the embedding.

2. Improved Privacy-Utility Tradeoff: The paper demonstrates that DPPN achieves a superior balance between privacy protection and maintaining the utility of the embeddings for downstream tasks.

3. Effective Against Real-World Threats: DPPN shows significant effectiveness in mitigating sensitive information leakage on real-world datasets, such as medical records and financial data.

weaknesses:
### 1. Personal Information Matching Issue

The DPPN method may not explicitly detail how to accurately match and process personal information. Ensuring accuracy and security, while protecting privacy, requires careful handling. This involves selecting and optimizing exact and fuzzy matching algorithms to provide accurate results without compromising privacy. The method uses a differentiable neuron mask learning framework to detect privacy neurons related to sensitive information. By assessing the importance of each embedding dimension, the top \( k \) dimensions with the highest importance are selected as privacy neurons. Visualizing the proportion of protected private data is necessary. The effectiveness of DPPN relies on accurately identifying and selecting neurons related to privacy information. Inaccuracies in this process may lead to sensitive information leakage.

### 2. Lack of Theoretical Guarantees

DPPN does not provide theoretical guarantees like Differential Privacy (DP), meaning it cannot quantify the degree of privacy protection or ensure the statistical insignificance of including or excluding a single data point. Adapting DPPN's targeted perturbation method to meet DP standards is challenging, as DP requires perturbing all published data for strong privacy guarantees, whereas DPPN only perturbs a subset of dimensions.

### 3. Challenges in Multilingual and Multicultural Backgrounds

Expressions of personal information vary across languages and cultural backgrounds. It is essential to discuss whether the DPPN method can adapt to and effectively handle these differences.

### 4. Real-Time Performance and Computational Cost

The real-time performance and computational cost of DPPN in practical applications are unclear. This is an important consideration for systems that need to process large volumes of data in real-time. The interpretability of the DPPN method is relatively low, potentially limiting its use in scenarios requiring high model interpretability, such as medical diagnosis.

### 5. Embedding Methods

The authors use embedding methods like GTR-base, Sentence-T5, and SBERT. It is suggested that traditional methods such as GloVe and Word2Vec be discussed and experimentally analyzed.

### 6. Presentation Issues

Table 9 is too large. Using \resizebox \textwidth is not recommended. Table 8: Statistics of datasets. The caption is below the table; the authors should unify the format. Authors can add more related work in the context:
- Private Language Models via Truncated Laplacian Mechanism EMNLP 2024
- Differentially Private Language Models Benefit from Public Pre-training PrivateNLP 2020

The explanations for Figures 1 and 3 are not detailed enough, especially regarding the meaning of arrows and labels. It is recommended that the authors clarify these in the legend.

questions:
1. What are the benefits of neuron mask learning and direct matching for replacing PII compared to constructing a series of regular expressions to match and transform PII in the corpus?
2. What are the benefits or advantages of the author's method compared to traditional DP-based methods?
3. Could the author fully describe the challenges faced in this research?
4. The author would be greatly appreciated if the code could be made open source and contributed to the community.
5. Six datasets were used in the experiments, but only two datasets' results are shown in the main text. Could the author explain in more detail the reasons for choosing these two datasets?
6. The results in Table 1 show the performance of different methods, but there is a lack of in-depth analysis of these results. It is suggested to add a discussion of the results, explaining why DPPN performs better in these cases.
</content>
</thread_object>

</thread>
<thread>
creation date:
1729748732067

thread_type:
official_review

<thread_object>
creator: reviewer_qccl

<content>
summary:
This paper introduces DPPN (Defense through Perturbing Privacy Neurons), a novel method that protects text embeddings by selectively identifying and perturbing privacy-sensitive neurons. The experiments show the effectiveness of the method.

strengths:
1. This paper has a clear format, highlighting important keywords throughout the paper.
2. The paper explains the methodology in details.

weaknesses:
1. Is LapMech a variant of DP? What is the performance comparison with standard DP? The paper can add a new section on this.
2. What is "Downstream" in utility metric? The paper could explain more about how they measure the utility of the method.
3. How do you explain the results in Table 7? Table 7 gives more results of the method than Table 1, and It seems that DPPN does not outperform other defense methods in some datasets.
4. The paper can include some other defense methods in their comparison. Experiments with two baselines are not very convincing.
5. In Sec 4.1, they mention that "Vec2text serves as our primary attack model in subsequent experiments." So it seems that only one attack model is evaluated in the whole paper, making the defense performance not convincing again. Are there any results for other attack models?
6. Only Table 3 shows some results for attack models. However, only privacy metrics are presented. How about the utility metrics?
7. The paper could reorganize some experimental results based on some comments above, making the experiment section better.

questions:
Please see those in weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730806331874

thread_type:
official_review

<thread_object>
creator: reviewer_gaog

<content>
summary:
This paper focuses on defense strategies against embedding inversion attacks, a type of privacy attack where attackers attempt to reconstruct original sensitive data from its embedding representation. Existing defense methods commonly add noise uniformly across all embedding features. However, this approach is limited in maintaining model performance and is limited in effectively protecting privacy since, ideally, more noise should be directed towards privacy-sensitive features.

To address these issues, the authors first assume and validate that embeddings are composed of both privacy-sensitive and privacy-invariant features. Then, they propose an optimization problem in which a differentiable mask is optimized to isolate privacy-sensitive information within learned embeddings. The optimized mask becomes a tool to detect privacy-sensitive features, and by adding noise to these features, the authors achieve defense against embedding inversion attacks.

strengths:
1. The method is technically sound, successfully enhances benign accuracy in downstream tasks, and prevents privacy leakage.

2. The hypothesis validated in this paper, that features can be divided into privacy-sensitive and privacy-invariant categories, is quite interesting. Building on this, the idea of using a mask to separate these features is also very novel.

weaknesses:
1. The paper lacks a formal privacy guarantee. For instance, in methods like LapMech, the authors provide a proof to demonstrate the effectiveness of privacy protection, but this paper lacks such a discussion. Such a drawback raises doubts about the trustworthiness of the proposed method, especially if attackers know the defense mechanism.

2. This defense method requires two datasets, one containing privacy-sensitive data and one without, which necessitates labeling what information as private information. This adds a labeling burden in real-world datasets, as additional annotation is required.

questions:
1. Is it possible to formulate privacy realized by this framework?

2. It is recommended that the authors consider adaptive attack scenarios to demonstrate that this defense method remains effective even if the privacy protection scheme is exposed to adversaries.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732210427586

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response on the theoretical guarantees of DPPN

comment:
We acknowledge that the absence of a formal theoretical privacy guarantee represents a limitation of our work. Below, we outline a promising direction for incorporating formal privacy guarantees into DPPN and discuss the associated challenges in achieving this goal.


**Extending DPPN with Regularized Mahalanobis Differential Privacy:**

To meet standard differential privacy requirements, all dimensions of the text embeddings must be perturbed rather than focusing on a subset of embedding dimensions. A potential approach involves introducing "elliptical" noise into the text embeddings. Unlike the traditional Laplace mechanism, which adds isotropic (spherical) noise sampled uniformly from a multivariate Laplace distribution, elliptical noise allows for dimension-wise perturbation with variable scales. The Mahalanobis distance allows for more flexible, dimension-wise perturbations by accounting for the covariance structure of data, rather than treating all dimensions equally. This technique has been studied in the context of word embeddings [1], where privacy is defined under a metric-local differential privacy (metric-LDP) framework using the Mahalanobis distance (Definitions 3 and 4 of [1]). While this direction is promising, several challenges must be addressed.
Technical Challenges: 
1) **Curse of Dimensionality in High-Dimensional Data:**  Differential privacy in high-dimensional spaces faces significant challenges, as the noise magnitude required for  privacy increases linearly with the embedding dimension [2]. Specifically, the noise norm in the Laplace mechanism follows the Gamma distribution $\Gamma(d, \epsilon)$, where $d$ is the embedding dimension and $\epsilon$ is the privacy budget.
This requires setting a very low privacy budget (e.g., $\epsilon \geq 500$) to maintain embedding utility for modern text embedding models with large dimensions (e.g., $d=768$ for SentenceBERT). While dimension reduction methods like Gaussian random projection could mitigate this issue, they are incompatible with DPPN's core design because they disrupt the separation of privacy-sensitive and privacy-invariant dimensions that is central to DPPN. 

2) **Deriving a Formal Proof for the Neuron-Suppressing Perturbation Function:**
As demonstrated in Sections 3.3 and 5.2, our perturbation function achieves effective defense against inversion attacks compared to existing DP-based mechanisms. However, constructing a formal proof that this approach satisfies the Regularized Mahalanobis differential privacy guarantee is non-trivial. Specifically, this would require defining the covariance matrix for the Mahalanobis distance and formulating a corresponding perturbation function that meets metric-LDP criteria. We acknowledge this as a critical avenue for future work and intend to explore formalizing these guarantees in subsequent research.

**Discussion:**

We believe the core contribution of our work—defending against inversion attacks through dimension-selective perturbation—addresses a critical gap in the literature. Moreover, it is worth noting that other prominent works, such as [3, 4, 5], have also made meaningful contributions to privacy research despite the lack of formal guarantees. We hope that our findings will inspire further exploration and innovation in embedding-based privacy defense mechanisms.

**References**

[1] Xu, Zekun, et al. "A Differentially Private Text Perturbation Method Using Regularized Mahalanobis Metric." Proceedings of the Second Workshop on Privacy in NLP. 2020.

[2] Wu, Xi, et al. "Bolt-on differential privacy for scalable stochastic gradient descent-based analytics." Proceedings of the 2017 ACM International Conference on Management of Data. 2017.

[3] Coavoux, Maximin, Shashi Narayan, and Shay B. Cohen. "Privacy-preserving Neural Representations of Text." Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. 2018.

[4] Elazar, Yanai, and Yoav Goldberg. "Adversarial Removal of Demographic Attributes from Text Data." Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. 2018.

[5] Struppek, Lukas, Dominik Hintersdorf, and Kristian Kersting. "Be Careful What You Smooth For: Label Smoothing Can Be a Privacy Shield but Also a Catalyst for Model Inversion Attacks." The Twelfth International Conference on Learning Representations.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This is a borderline paper. While the reviewers agreed on the merits of the problem, and some of the algorithmic approaches, there were legitimate concerns about the lack of a formal privacy guarantee. Since it seems methods developed in the differential privacy literature is applicable, so a question that is glaring is how the current method compare to formal privacy preserving methods. The authors did summarize it as a future direction. It seems that having the formal privacy component is crucial to the paper.