Stealthy Shield Defense: A Conditional Mutual Information-Based Approach against Black-Box Model Inversion Attacks

Authors: Tianqu Zhuang, Hongyao Yu, Yixiang Qiu, Hao Fang, Bin Chen, Shu-Tao Xia
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: AI security, model inversion attack, information bottleneck, conditional mutual information

Abstract:
Model inversion attacks (MIAs) aim to reconstruct the private training data by accessing the public model, raising concerns about privacy leakage. Black-box MIAs, where attackers can only query the model and obtain outputs, are closer to real-world scenarios. The latest black-box attacks have outperformed state-of-the-art white-box attacks, and existing defenses cannot resist them effectively. To fill this gap, we propose Stealthy Shield Defense (SSD), a post-processing algorithm against black-box MIAs. Our idea is to modify the model's outputs to minimize the conditional mutual information (CMI). We mathematically prove that CMI is a special case of Information Bottleneck (IB), and thus inherits the benefits of IB---making predictions less dependent on inputs and more dependent on ground truths. This theoretically guarantees our effectiveness, both in resisting MIAs and preserving utility. To minimize CMI, we formulate a convex optimization problem and solve it via the water-filling method. Without the need to retrain the model, our defense is plug-and-play and easy to deploy. Experimental results indicate that SSD outperforms existing defenses, in terms of MIA resistance and model's utility, across various attack algorithms, private datasets, and model architectures. Our code is available at https://github.com/ZhuangQu/Stealthy-Shield-Defense.

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730184309635

thread_type:
official_review

<thread_object>
creator: reviewer_j1l4

<content>
summary:
This paper addresses the critical issue of protecting machine learning models from black-box model inversion attacks (MIAs). Specifically, the attacker’s objective is to reconstruct private training data by only accessing the model's outputs.The authors therefore propose a novel defense mechanism named Stealthy Shield Defense (SSD) to post-process the model’s output such that the information revealed by the model’s output about the private data is minimized under constrain. This method leverages Conditional Mutual Information (CMI) to reduce the dependency between the model's output and private data. In addition, they also propose an adaptive rate-distortion framework using water-filling method to preserve the utility while minimizing the CMI.

strengths:
1. The authors propose SSD, a post-processing defense mechanism that doesn’t require retraining the model, making it practical for real  deployment. 

2. The authors theoretically prove that minimizing CMI serves as a special information bottleneck, therefore minimizing CMI can effectively balance data privacy and utility. By iterating the CMI through all possible labels, the whole dataset can thus be. 

3. The paper introduces an adaptive rate-distortion mechanism optimized using the water-filling method. This approach efficiently calibrates the probability distributions output by the model. 

4. The authors validate their method across various attack algorithms (BREP, Mirror, C2F, LOKT), datasets (FaceScrub, CelebA), and model architectures (VGG-16, IR-152). The results demonstrate that SSD outperforms existing defenses in terms of defense scheme robustness and effectiveness on preservation of model utility.

weaknesses:
1. Even though the authors claim that the computational overhead is negligible due to the efficient optimization on GPU, a more detailed analysis or benchmarking of the computational cost would greatly support this claim. 

2. Using MI/CMI in deep learning is gaining increasing attention. However, its introduction in related work lacks both depth and breadth, which makes it hard to find the role of this work into the relevant community. 

3. The authors should use a deeper model architecture and higher-dimensional input data for training and prove the effectiveness of the proposed method. When the input data is in high dimensionality, it usually contains a significant amount of irrelevant information. Even worse, since the model depth is also high, the final output, Y ̂, may only contain a small amount of MI with the input, X. I am wondering if optimizing I(X;Y ̂ ∣ Y) will be challenging in this case. 

4. Even though this is a robust scheme against model inversion attack, the authors should discuss about the potential possibility of adaptive attacks. If the adaptive attack is unlikely to happen for now, the authors should also state the reason why.

questions:
1. The proposed scheme requires iterating over all possible labels to perform the defense. What happens if there are a large number of possible labels? For example, a large company or region may train a face recognition model that includes thousands or millions of faces. Will this scale slow down the defense and become a bottleneck? I suggest some detailed computational analysis.

2. As the authors pointed out in the paper, X→Z→Y ̂ is a Markov chain. It is possible that when input data has large dimensionality, or the model has very deep layers. Under this circumstance, it is likely that Y ̂ shares very little information with X. The derived MI I(X;Y ̂ ∣ Y) will have such a small value that could be hardly optimized. Does the proposed scheme still hold effective in this scenario? The authors should provide more empirical/theoretical analysis to justify this. 

3. Are there any manifest adaptive attack schemes that can target at this defense? How difficult is it to design/launch such attacks?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730327058380

thread_type:
official_review

<thread_object>
creator: reviewer_ksss

<content>
summary:
This paper proposes a post-processing based defense to protect against black-box model inversion attack. The key insight of the paper is to transform the output of the model to reduce conditional mutual information (CMI). This is done to reduce the dependence between the output and the input while preserving the dependence on the true labels. The authors develop a mechanism to transform the model’s prediction to reduce CMI by framing it as an optimization problem. The experimental results on multiple image classification datasets show that their proposed SSD defense provides a better privacy-utility tradeoff compared to prior defenses.

strengths:
1. The paper is well-written and easy to follow.
2. The idea behind the proposed defense is intuitive and presented well.
3. Experiments seem comprehensive and the defense shows better utility-privacy tradeoff compared to prior defenses in both soft and hard label settings
4. The defense is post-processing based, making it easy to adopt.

weaknesses:
1. The proposed defense could be susceptible to an adaptive attack. An adversary could query the same input multiple times to obtain multiple predictions from the model. Since the defense produces outputs by perturbing the original prediction, the adversary could compute an average over multiple outputs to get a better estimate of the model’s true output. Such an adaptive attack is not discussed by the paper.
2. The defense requires a validation dataset to implement, which could limit its adoption.

questions:
1. In line 228, the authors state that “the objective function is too complex for the convex optimizer to solve.” and use this as the motivation to minimize a simplified objective $KL(p||q^y)$ instead (by sampling $y\in\mathbb{Y}$). Why is this the original objective too complex? Wouldn’t you be able to use gradient descent to solve for $p$? Would this lead to a better solution?
2. How was the temperature $T$ picked in Algorithm 1?
3. What was the value of $\epsilon$ for the proposed defense in the experiments?
4. Why is the Acc@1 for IR-152 lower for “no defense” compared to prior defenses? This suggests that adding prior defense improves the attack success rate, which is very strange.
5. Can you address the adaptive attack discussed in Weakness#1?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730638827981

thread_type:
official_review

<thread_object>
creator: reviewer_ui99

<content>
summary:
This paper proposes a post-processing defense SSD based on conditional mutual information (CMI) especially for defending against black-box model inversion attacks. It theoretically proves that CMI is effective and further reduce modifications to outputs by proposing adaptive rate-distortion framework. Experiment results indicate that SSD achieves SOTA better utility-privacy trade-off.

strengths:
-	This paper considers defense against model inversion attack from a new perspective by proposing CMI, which provides new insights into this field.
-	It provides thorough theoretical analysis to support the effectiveness of the SSD defense, with a clear and rigorous logical structure.
-	The paper achieves state-of-the-art performance in terms of utility-privacy trade-off and defense against advanced black-box attacks, qualitatively demonstrating its effectiveness.

weaknesses:
-	Key concepts, such as the water-filling method, lack adequate coverage in the main body, making it challenging for readers to fully grasp the methodology without delving into appendices. Greater emphasis on these aspects within the main text would improve the paper's accessibility.
-	The experimental setup omits certain critical information, such as the dataset used for GAN prior training and specifics for Figure 1, leaving questions regarding reproducibility and completeness.
-	The study exclusively benchmarks against state-of-the-art white-box defenses but omits comparisons with black-box defenses, which would be more pertinent for the black-box MIA context.


Minor remarks:

-	There are also minor errors, such as duplicated words (line 107), inconsistent notation (duplicate use of $p$ at line 215) and typo at line 314.
-	The notation is unconventional, such as using non-bold font for vector inputs like $\mathbf{x}$, which reduces consistency with standard notation practices.

questions:
-	How does SSD perform when the distribution of the public dataset differs significantly from the private dataset (e.g., public dataset = FFHQ, private dataset = CelebA)? Clarification on this point would help to understand SSD’s adaptability under various deployment conditions.
-	The paper states that SSD shows exceptional performance in reducing attack accuracy for LOKT and BREP methods. Could the authors elaborate on why SSD is particularly effective against these specific methods?
-	Could the authors provide additional details on the GAN prior training and any specific hyperparameters used for Figure 1?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730687086574

thread_type:
official_review

<thread_object>
creator: reviewer_vqzg

<content>
summary:
This paper proposes a post-process black-box MI defense Stealthy Shield Defense (SSD) based on conditional mutual information (CMI). By leveraging CMI, SSD aims to reduce the model's output dependence on its input. The experiments demonstrate SSD's effectiveness against four state-of-the-art black-box MI attacks on two datasets.

strengths:
* This work focuses on defending against black-box MI attacks which have not been addressed in the existing defense.

* As a post-processing defense, SSD can be easily integrated with most pre-trained models for defending against black-box attacks.

* Empirical results demonstrate SSD's success in reducing the attack performance of four state-of-the-art black-box MI attacks.

weaknesses:
* Practical Limitations: Although the idea of post-processing defense is interesting, the proposed method raises concerns about its applicability in real-world scenarios. To modify the model's prediction output, **SSD requires a dataset $D_{valid}$**, which I believe should be real data (either the training dataset or its validation set). This means the user must store raw training data or predictions on the training data to perform predictions, potentially increasing the risk of data leakage.

* SSD's prediction process involves an optimization step for each image, leading to significantly increased computational costs and slower inference times compared to other models.

* The experiments were conducted on low-resolution 64x64 images, limiting the generalizability of the findings high-resolution scenarios.

* [r1] was omitted in the paper while it is also a state-of-the-art black-box attack.

* The paper suffers from some typos and grammatical errors. For examples, lines 41, 107.

[r1] Han, Gyojin, et al. "Reinforcement learning-based black-box model inversion attacks." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

questions:
* [r1] was omitted in the paper. I suggest the author should add [r1] in the experiments

* Can the author evaluate the proposed method's performance on high-resolution images? Additionally, please provide more details on how high-resolution MI attacks like MIRROR are adapted to low-resolution scenarios.

* Algorithm 1 references $D_{valid}$ without a clear definition. Please provide an explicit expression for $D_{valid}$. Furthermore, it would be interesting to understand the impact of $D_{valid}$’s size on the final output. Were all training images used as $D_{valid}$?

* For the LOKT experiments, please clarify whether the authors retrained their TACGAN and other surrogate models or used pre-trained models. Note that LOKT requires training target models with GANs and surrogate models,  which in this case SSD needs to be performed on every samples.

*  Please specify the number of identities used for attacks and the quantity of reconstructed images generated per identity.

* I would like to see a comparison of the prediction time between SSD and baseline methods like NoDef or other defenses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732227913864

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Common Concerns on Computational Cost

comment:
We sincerely appreciate all the reviewers for dedicating their valuable time and effort to review our paper. Their insightful comments and suggestions are conducive to improving our paper.

Encouragingly, the reviewers praise the significance of black-box defense (R#VqZg), the generalizability of our post-processing (R#VqZg, R#KsSS and R#J1L4), the insight of using conditional mutual information (R#Ui99, R#KsSS and R#J1L4), the rigor of our mathematical proof (R#Ui99 and R#J1L4), and the effectiveness shown by our experiments (all reviewers).

Below, we uniformly address the concerns shared by more than one reviewers.

---

## Common Concerns on Computational Cost (R#VqZg and R#J1L4)

We formulate the minimization of CMI as a convex optimization problem. Our post-processing module needs to solve it for each prediction, which raises concerns about computational overhead. We indeed encountered this problem before. At that time we turned to CVXPY [1], a famous convex optimizer, but its efficiency and accuracy are not satisfactory. Fortunately, this concern has been solved through our efforts:

**Theoretically, we propose to solve the convex optimization problem via water-filling.** After a series of mathematical derivations (presented in appendix), we find the explicit form of the optimal solution. Our water-filling algorithm can reach the solution with the time complexity of $O(|\mathbb{Y}|\log|\mathbb{Y}|)$, where $|\mathbb{Y}|$ is the number of labels.

**Practically, we implement our water-filling on GPUs.** We manage to replace the branching and looping operations with sequential vectorization operations, and then implement our algorithm on PyTorch Tensors (provided in supplementary material). Our GPU-based-water-filling can be regarded as a post-processing layer of neural networks, whose computational cost is not greater than the other layers.

Following the reviewers' suggestions, **we add a quantitative experiment.** The models, datasets, and defense settings are the same as the Table 3 in our paper. We take a batch with 512 samples and let the model infer 100 times on it. We record the time cost by `torch.profiler`, an official tool provided by PyTorch. We exclude the time for I/O (i.e. the time from disk to memory, and from CPU to GPU), and only include the time for forward propagation on GPU. Our experiment is conducted on one NVIDIA GeForce RTX 3090. Here is the result:

|                           | IR-152 & CelebA | IR-152 & FaceScrub | VGG-16 & FaceScrub |
| :-----------------------: | :-------------: | :----------------: | :----------------: |
|   Time without defense    |     18.63 s      |       17.70 s       |       5.65 s        |
|   Time with our defense   |     19.22 s      |       18.16 s       |       6.07 s        |
| Percent of increased time |      3.1%       |        2.5%        |        7.4%        |

It can be seen that we only increased the time by 2.5% to 7.4%. The higher percent on VGG is due to the simpler model structure. In absolute terms, **modifying 512 predictions for 100 times only needs 0.5 seconds. If we take the I/O time into account, the percents are small enough to be ignored.** We will add this experiment to the appendix.

---

[1] Steven Diamond and Stephen Boyd. CVXPY: A Python-embedded modeling language for convex optimization. Journal of Machine Learning Research, pp. 1–5, 2016.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732229988761

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Common Concerns on Adaptive Attacks

comment:
We sincerely appreciate all the reviewers for dedicating their valuable time and effort to review our paper. Their insightful comments and suggestions are conducive to improving our paper.

Encouragingly, the reviewers praise the significance of black-box defense (R#VqZg), the generalizability of our post-processing (R#VqZg, R#KsSS and R#J1L4), the insight of using conditional mutual information (R#Ui99, R#KsSS and R#J1L4), the rigor of our mathematical proof (R#Ui99 and R#J1L4), and the effectiveness shown by our experiments (all reviewers).

Below, we uniformly address the concerns shared by more than one reviewers.

---

## Common Concerns on Adaptive Attacks (R#KsSS and R#J1L4)

Reviewers are concerned about adaptive attacks, where attackers know our defense and specifically design their attack strategies. We appreciate them for raising such an interesting question, which has stimulated our deeper thinking.

Firstly, we believe that launching adaptive attacks in black-box scenarios is unrealistic, because attackers don't know the target model, and naturally don't know its defense strategy. If they were to guess the defense strategy based on the model's behavior, they would need to consume a large number of queries.

Step back and consider, **if attackers know our defense, the best attack strategy we think is:**
1. Query the same $\pmb{x}$ repeatedly and count the frequency of different outputs.
2. Estimate our sampling probability $\mathbb{P}(y|\pmb{x})$ by the frequency.
3. Infer our true prediction $\mathbb{P}(\hat{y}|\pmb{x})$ based on $\mathbb{P}(y|\pmb{x})$ and temperature $T$ (assuming they know).

If an online server detects such pattern of queries, it can block them. Step back and consider again, **we now propose a memory-free and low-cost improvement against adaptive attacks:**

Design a hash function $h:\mathbb{X \rightarrow N}$, where $\mathbb{X}$ is the input space and $\mathbb{N}$ is the set of integers. When users/attackers query $\pmb{x}$, we take $h(\pmb{x})$ as a random seed for sampling on the probability $\mathbb{P}(y|\pmb{x})$. This ensures *same-input-same-output*, thus preventing the adaptive attackers. However, attackers can add subtle perturbations to $\pmb{x}$, therefore our $h$ needs to be robust. For example, it can be
$$ h(\pmb{x}) \coloneqq \sum_{i = 1}^{m}\lfloor k \cdot z_{i}(\pmb{x}) \rfloor $$
where $\pmb{z}(\pmb{x}){\in}\mathbb{R}^{m}$ is the penultimate layer feature in target model, and $k$ is the sensitivity coefficient. Note that $\pmb{z}(\pmb{x})$ are commonly used to assess the similarity between different images, i.e., **the closer the two $\pmb{z}(\pmb{x})$ are, the more similar the two $\pmb{x}$ look.** We map $\pmb{z}(\pmb{x})$ to integers by rounding down and summing up. The larger $k$ is, the more numerically sensitive $h$ is, and the more random our defense is.

How to evaluate and improve $h$ is a new topic, and we believe it can be an independent research work. Thanks again to the reviewers for pointing out adaptive attacks, and we will supplement the discussion above in our paper.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper presents a new defense against model inversion attacks. The main idea is to post-process the model's output by minimizing the conditional mutual information between the predicted probability vector and the input. The authors show experimentally that their defense can effectively mitigate state-of-the-art black-box attacks. While reviewers raised several weaknesses (see below), most of them are addressed by the rebuttal, and AC is happy to recommend acceptance conditioned on these changes being included in the camera ready version.