Spurious Privacy Leakage in Neural Networks

Authors: Chenxiang Zhang, Jun Pang, Sjouke Mauw
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: spurious correlation, membership inference, privacy, robustness, safety

Abstract:
Neural networks are vulnerable to privacy attacks aimed at stealing sensitive data. When trained on real-world datasets, these models can also inherit latent biases, which may further increase privacy risks. In this work, we investigate the impact of spurious correlation bias on privacy vulnerability, identifying several key challenges. We introduce _spurious privacy leakage_, a phenomenon where spurious groups can be up to 100 times more vulnerable to privacy attacks than non-spurious groups, and demonstrate how this leakage is connected to task complexity. Furthermore, while robust training methods can mitigate the performance disparity across groups, they fail to reduce privacy vulnerability, and even differential privacy is ineffective in protecting the most vulnerable spurious group in practice. Finally, we compare model architectures in terms of both performance and privacy, revisiting prior research with novel insights.

Decision: reject
Reviewer scores: 5, 8, 5, 3

Reviews and discussion:
<thread>
creation date:
1729574907365

thread_type:
official_review

<thread_object>
creator: reviewer_9gk7

<content>
summary:
The paper examines how bias in neural networks affects privacy vulnerabilities, particularly focusing on spurious correlations. The paper has several findings after experiments.

strengths:
The spurious privacy leakage in neural networks is an interesting topic and the paper has a good format emphasizing each finding after some experiments.

weaknesses:
1. Current research has much more literature than what the paper presented in the related work section.
2. The authors should explain why these datasets are chosen and not the other datasets.
3. In Sec 3.1, the authors mention that "The largest privacy disparity is observed at ~3% FPR area of Waterbirds, where the samples in the most spurious group are ~100 times more vulnerable than samples in the non-spurious group." Where do "3%" and "100 times" come from? The figure does not clearly show this and the paragraph does not explain this. 
4. The claim of "spurious groups can be up to 100 times more vulnerable to privacy attacks than non-spurious groups" is just a result from one data point. This is exaggeration and should not include in the abstract.
5. While the paper presents experiments for its findings, the limited number of experiments conducted for each conclusion raises questions about results. Additional experiments are needed for each finding.
6. Finding V claims that they draw the conclusion under a fair experimental setup. However, the authors use different hyperparameter settings for different models, which is not a fair comparison.

questions:
See weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1729575729920

thread_type:
official_review

<thread_object>
creator: reviewer_e78d

<content>
summary:
The authors uncover that spurious groups in datasets (i.e., groups affected by spurious correlations) are significantly more vulnerable to membership inference attacks (MIA) than non-spurious groups. Meanwhile, as the task complexity decreases (e.g., fewer target classes in the dataset), the privacy leakage for spurious groups remains constant or worsens, while leakage for other groups reduces. Moreover, despite improvements in group performance disparity through methods like DRO, DFR, and DP, these methods fail to address the privacy disparity among spurious groups. They also show that architectures like Vision Transformers (ViTs) are not necessarily more robust against spurious correlations than convolutional models under fair experimental setups.

strengths:
- The paper focuses on the connection between spurious correlations and privacy leakage, an underdeveloped topic in trustworthy machine learning.
- The paper presents several interesting observations.
- The paper is well-organized and easy to follow.

weaknesses:
1. The motivation behind evaluating privacy disparities among subgroups (spurious vs. non-spurious groups) is unclear. While the paper shows that existing methods (DRO, DFR, or DP-SGD) may not fully address these privacy gaps, it's unclear why privacy parity across subgroups could be a priority. Why should we care about these gaps?

2. While the authors present technically interesting observations about privacy disparities, the results are more like experimental reports. It’s unclear how these findings can contribute to the field. For instance, could they inspire new defense strategies? Or inform advanced attack methods? 

3. The conclusions in Sections 3 to 5 are important but not well-supported as they rely on limited experiments, datasets, and methods, which is not convincing. For example, Section 4.2 states that DP fails to protect certain vulnerable groups in the data. The results contradict previous research suggesting that DP can protect sample vulnerabilities by preventing memorization. The finding comes from a single experimental setup. The scope is too narrow to fully support such a broad conclusion. More ablation studies are needed. 

4. The paper needs more detailed discussions and comparisons to better support the conclusions. It’s unclear how these findings are connected to current studies. For example, in Section 3, there is related work on subgroup evaluations of model fairness and privacy with MIA—are the findings consistent? For Section 4, previous studies have explored the utility tradeoffs of DP methods—do they also show a failure to protect samples? Similarly, in Section 5, prior research compares the model robustness of Vision Transformers and CNNs—do those results align? More discussion is needed across these sections to connect the findings with existing work.

5. The related work section feels too limited, given the paper covers multiple topics.

questions:
see above
</content>
</thread_object>

</thread>
<thread>
creation date:
1730295477094

thread_type:
official_review

<thread_object>
creator: reviewer_kin4

<content>
summary:
This paper shows experimentally that in data with spurious correlations and imbalanced groups, the minority groups are more susceptible to membership inference attacks than the majority groups. This high level message -- that out-of-distribution examples tend to be less private -- was known before: see for example, Figure 13 in the LiRA paper, [1], where this fact is exploited to design better privacy tests, as well as [2]. However what I like about this paper is that they do a very comprehensive experimental study on real data, and show a number of additional conclusions, and hence there is value in accepting it. 
[1] https://arxiv.org/abs/2210.02912
[2] https://arxiv.org/abs/2202.05189

strengths:
1. The paper presents highly comprehensive experiments that are quite well done. The results are well-presented, and well-supported by experimental evidence.

weaknesses:
1. I would urge the authors to make some of the details a little bit more transparent in the main body. One difference between the standard setting and this work is that MIAs are used for fine-tuning and not pre-training data. This may mean that the fine-tuned datasets are very small per model. One of the best-kept secrets in LIRA-style membership inference is that the MIA is always carried on models that are trained on only a subset of the data, and making that subset bigger leads to worse "privacy loss". 

Since this paper is further using MIA on models fine-tuned on a small amount of data, what is the size of the data that the model is fine-tuned on? 

2. I am also not sure how meaningful the differential privacy results are -- since here epsilon=128. That kind of value for epsilon offers quite negligible privacy. That being said the remaining results are quite interesting.

questions:
See Weakness 1.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730687041783

thread_type:
official_review

<thread_object>
creator: reviewer_7w3c

<content>
summary:
This paper investigates the finding that groups influenced by spurious correlations in datasets are more vulnerable to membership inference attacks (MIA) than other groups. The study also shows that Vision Transformers (ViTs) are not necessarily better than convolutional models at handling spurious correlations. The paper highlights how bias in neural networks affects privacy risks.

strengths:
1. The paper attempts to undermine the relationship between bias an d privacy leakage. The key findings are well articulated via examperiments. 
2. The paper is overall well written and easy to follow.

weaknesses:
1. The motivation is not very clear. What is the reason to assess privacy disparities between spurious and non-spurious subgroups?
2. The related work section is too brief considering the multiple topics the paper covers. Expanding it to include more of the relevant literature would strengthen the foundation. It is also unclear how the findings align with current research.
3. The experiments are not clearly explained. Including the choice of datasets and neural network models.

questions:
Refer to the weakness section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731851812207

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Overall Comment by Authors

comment:
The following paragraphs collect the answers to a few common doubts raised by different reviewers.

__1) Expanded literature__

Our literature study is focused on 3 different topics: 1) membership inference, 2) spurious correlation, 3) papers with a higher-level perspective, related to privacy and safety ML. For pedagogical reasons, we decided to divide these three topics into three different (sub)sections, as close as possible to the relevant part of the paper. The topics 1 and 2 are in the background section as they provide the notions for understanding the work, while topic 3 is discussed in the related work section. Together, these sections currently include ~25 citations. Further, in response to the feedback, we will expand the sections with additional related works that enhance the context and additional discussion with prior works. Lastly, if the reviewers have any specific work that we should have cited in mind, we will follow your suggestions and add them to the right spot in our paper. 

__2) Additional results__

We add new experiments for the section 4.2 focused on differential privacy:

We train DP-SGD target models on CelebA and FMoW for 50 epochs using batch size 1024 and grid searching the lr in [1e-1, 1e-2], \epsilon in [1, 2, 8, 32, 128], and \delta of 1e-5. We reuse the shadow models trained in section 3.1 for LiRA. Consistent with our findings for Waterbirds, low ϵ (=1, 2) hinders the generalization, while high ϵ better mitigates spurious correlations. However, in terms of privacy vulnerability, all models across different privacy budget levels exhibit similar levels of vulnerability, highlighting the challenges of applying DP while retaining the utility of spurious data.

Table 1. Models trained on the FMoW dataset fail to generalize under a low privacy budget and are therefore ignored. The privacy vulnerability remains similar across different levels.
| eps | delta  | test acc (avg)  | test acc (worst) | tpr\@0.1%fpr (avg) | tpr\@0.1%fpr (worst) |
|-----|--------|-------|-----------|------|------|
| 1   | 1e-5   | 13.01 | 9.45      | - | - |
| 2   | 1e-5   | 19.58 | 12.53     | - | - |
| 8   | 1e-5   | 38.64 | 26.34     | 6.66 | 6.47 |
| 32  | 1e-5   | 43.58 | 27.77     | 6.83 | 6.60 |
| 128 | 1e-5   | 43.95 | 29.73     | 6.82 | 6.60 |

Table 2. Models trained CelebA dataset are more spurious robust as the privacy budget is relaxed. However, the privacy level remains similar across budgets. 
| eps | delta  | test acc (avg)  | test acc (worst) | tpr\@0.1%fpr (avg) | tpr\@0.1%fpr (worst) |
|-----|--------|--------|-----------|------|------|
| 1   | 1e-5   | 91.45  | 25.56     | 0.77 | 4.38 |
| 2   | 1e-5   | 92.79  | 40.00     | 0.75 | 4.38 |
| 8   | 1e-5   | 95.46  | 35.56     | 0.75 | 4.22 |
| 32  | 1e-5   | 95.64  | 41.11     | 0.77 | 4.06 |
| 128 | 1e-5   | 95.54  | 47.22     | 0.76 | 4.38 |

For the other sections, we argue that our experiments sufficiently support the findings. For section 3.1, our MIA setting includes four datasets with real-world spurious correlations, different target complexity, and diverse modalities. For section 4.1, the setting compares three different training methods on four datasets (12 different experiment setups). For section 4.2, we added the new results as described above. Lastly, for section 5, we fairly compare eight different model architectures, finding one sufficient counterexample to warrant revisiting the claim from Ghosal & Li (2024).
</content>
</thread_object>

</thread>
<thread>
creation date:
1732026026693

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Rebuttal Revision

comment:
First of all, many thanks to all the reviewers for your valuable feedback. We have improved our manuscript with the following changes:

- Add two new experiments for section 4.2
- Expand the related work section
- Expand the discussions with past research in sections 3, 4, 5
- Motivate the importance of our research
- Clarify the choice of datasets and models
- Clarify the results in section 3.1
</content>
</thread_object>

</thread>
<thread>
creation date:
1733307363415

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We provide a summary of this rebuttal period.

Reviewers find the topic of "spurious privacy leakage" interesting (9gk7, E78d, Kin4). In particular, reviewer Kin4 states that our results are well-presented and well-supported by experimental evidence. Reviewer E78d also finds our results interesting, and reviewer 7w3C notes that they are well-articulated via experiments. Moreover, all reviewers praised the presentation style. During the rebuttal phase, we addressed all concerns raised by the reviewers which can be summarized into three main points:

- __More experiments are needed (9gk7, E78d).__
We believe that the quality of datasets matters more than their quantity. For example, experiments based on CIFAR10/100 and ImageNet are expected to yield similar results due to their similarity. In contrast, our experiments are based on diverse real-world datasets, covering different tasks, modalities (vision and text), and target complexity (2, 3, 62 classes). Consistent observations across these diverse datasets provide even stronger evidence to support our claims. Additionally, during the rebuttal, we added 2 more experiments to support finding 4 (as requested by E78d). 

- __More literature discussion is needed (9gk7, E78d, 7w3C)__.
For pedagogical reasons, we note that some aspects of related work are included in the background section. Additionally, we expanded the related work section and incorporated discussions of prior studies into the results of each section.

- __Clarification on the motivation (E78d, 7w3C)__.
We clarified the motivation of our work in the introduction.

Many thanks to everyone for your service!
</content>
</thread_object>

</thread>


Metareview:
metareview:
The authors investigate the impact of spurious correlation bias on privacy vulnerability. They introduce a phenomenon called, spurious privacy leakage, and report that spurious groups can be more vulnerable to privacy attacks than non-spurious groups.

Reason for decision: I found this paper's main message a repetition of a relatively well known phenomenon: out-of-distribution examples tend to be less private (see examples given by Reviewer Kin4). What is valuable is that the authors provided comprehensive experimental study for this message. However, perhaps this is not such a strong or surprising message to most of the privacy/ML audiences.