Learning to Generate Inversion-Resistant Model Explanations
Authors: Hoyong Jeong
Suyoung Lee
Sung Ju Hwang
Sooel Son
Venue: NeurIPS 2022 Accept (2022)
Keywords: model inversion defense
model explanation
explainable AI

Abstract: The wide adoption of deep neural networks (DNNs) in mission-critical applications has spurred the need for interpretable models that provide explanations of the model's decisions. Unfortunately, previous studies have demonstrated that model explanations facilitate information leakage, rendering DNN models vulnerable to model inversion attacks. These attacks enable the adversary to reconstruct original images based on model explanations, thus leaking privacy-sensitive features. To this end, we present Generative Noise Injector for Model Explanations (GNIME), a novel defense framework that perturbs model explanations to minimize the risk of model inversion attacks while preserving the interpretabilities of the generated explanations. Specifically, we formulate the defense training as a two-player minimax game between the inversion attack network on the one hand, which aims to invert model explanations, and the noise generator network on the other, which aims to inject perturbations to tamper with model inversion attacks. We demonstrate that GNIME significantly decreases the information leakage in model explanations, decreasing transferable classification accuracy in facial recognition models by up to 84.8% while preserving the original functionality of model explanations.

Learning to Generate Inversion-Resistant Model Explanations
Hoyong Jeong, Suyoung Lee, Sung Ju Hwang, Sooel Son
KAIST (yongari38, suyoung lee, sjhwang82, sk.son) kr
Abstract
The wide adoption deep neural networks (DNNs) in mission-critical applications has spurred the need for interpretable models that provide explanations of the model's decisions. Unfortunately, previous studies have demonstrated that model explanations facilitate information Icakage, rendering DNN models vulnerable model inversion attacks. These attacks enable the adversary to reconstruct original images based on model explanations, thus leaking privacy-sensitive features. To this end, we present Generative Noise Injector for Model Explanations (GNIME), novel defense framework that perturbs model explanations to minimize the risk model inversion attacks while preserving the interpretabillties of the generated explanations. Specifically, we formulate the defense training minimax game between the inversion attack network on the one hand, which aims invert model explanations, and the noise generator network the other, which aims to inject perturbations to tamper with model inversion attacks. We demonstrate that GNIME significantly decreases the information leakage in model explanations, decreasing transferable classification accuracy in facial recognition models by up 84.8 44.8% while preserving the original functionality of model explanations.
two-player
Introduction
The recent widespread adoption of deep neural networks (DNNs) in building practical real-world applications has led to a surge of interest in explainable AI (XAI) since mission-critical tasks (e.g.. medical diagnosis) often request explanations of the model's predictions. Such explanations are not only helpful in debugging the model but also in providing explanations to end-users on how predictions are made. They are often provided the form of feature attribution map, which shows the importance of each feature computed based on the gradients of target model.
Unfortunately, recent studies have demonstrated that such explanations cause additional information leakage, which can be exploited by an adversary [1] 8, 19, 22, 32). Specifically, Zhao et al. [32] presented novel black-box model inversion (MI) attack which reconstructs an input image given its prediction vector and the model explanation from a target DNN model. They demonstrated that additional information leaked through model explanations (e.g.. Grad or Grad-CAM increases the performance of MI attacks by factor of 2.4 in terms of transferable classification accuracy, enabling high-fidelity attacks.
Figure coompares the result of MI attacks using model predictions alone (PredMI) and explanation- aware MI attacks using model predictions and explanations together (ExpMI). The examples re- constructed via ExpMI show high-quality faces compared to those via PredMI, leaking additional information, such facial expressions and accessories. However, previous works have only proposed the attacks exploiting the information leakage, without providing any defenses against them.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Thus, to defend against MI attacks exploiting model explanations, we propose GNIME, which generates eversion-resistal model explanations. Figure 2 shows the overall GNIME pipeline. GNIME consists of two phases: training and deployment. In the training phase, GNIME trains noise generator that injects perturbations into the explanations provided by the model under protection to minimize the MI threat. To train this noise generator, GNIME trains an MI attack network to compete against each other. Then, in the deployment phase, we deploy the trained noise generator to obfuscate the explanations generated for each test instance.
The key idea of our approach to establish two-player minimax game between the MI attack (INV) network and the noise generator (NG) network. We design the NG network to inject perturbations into the model's explana- tions (i.e., attribution map); the noise-injected model explanation is then fed into the INV net- work to reconstruct the inverted image. GNIME conducts an alternating training of these net- works. During the NG's 's turn, the NG network is trained to tamper with the inversion performance of the INV network while minimizing the pertur- bations to the model explanations. During the INV's turn, train the INV network to max- imize its model inversion performance. These alternating training procedures improve the INV network to better invert the given model's expla- nations while the NG network learns inject perceptually imperceptible inversion-resistant perturbations to the provided explanations. Af- ter training is complete, the GNIME owner de- ploys the NG network to inject perturbations into the original explanations before releasing them.
Target Image
PruMi
ExphI
ExpMI GNIME
Figure 1: Qualitative comparison of the face re- construction results of PredMI and ExpMI attacks using LRP explanations. The last column shows the ExpMI attack results when target ML system is under protection with GNIME.
We evaluate the efficacy of GNIME, by measuring 1) how effective the model is in decreasing the risk of information leakage from the model explanations and 2) how well the perturhed explanations preserve the semantics of the original explanations. For different types of explanations (i.e., Grad (24), Grad-CAM (11) and LRP ()), we demonstrate that GNIME consistently generates infinitesimal inversion-resistant noises that significantly impede explaaaiio--aware MI attacks. To facilitate further
research, we publish GNIME at
httgggtttuu..... www.hhu...ooooooaooeeeeeeeaeeeee www.o///SS--. LAB/GMME
In summary, our contributions are as follows:
We propose the very first defense framework against explanation-aware MI attacks, which learns to suppress inversion-criticn features from model explanations.
We evaluate GNIME on diverse datasets and demonstrate its efficacy in decreasing the MI risk while preserving the interpretability of the original explanations.
2 Related Work
Explainable AI for CNN models. Previous XAI research has introduced diverse methods of computing an attribution map (i.e., saliency map) that explains the decision of DNN classifier. Specifically, given an input image, its attribution map highlights important features that heavily influence the classifier's decision. A straightforward approach to calculate the gradient of the DNN model's output with respect to each input pixel (24). which we refer to as Grad. Grad thus produces an attribution map of which the size is identical to that of input images. The larger the absolute Grad value of pixel, the greater influence the corresponding pixel has in deriving a model's decision. Numerous variants have been proposed to supplement or extend the explainability of Grad [3 25, 26) In this paper, we use Grad as basic type of for model explanations. The class activation map (CAM) [3] is also a prevalent means of computing model explanations. provides a coarse grained activation map that represents the state of intermediate convolutional layers. Gradient- weighted CAM or Grad-CAM 21 is an extension of CAM. Unlike CAM, Grad-CAM does not require specific model architecture. It performs partial propapropngttion until reaching the final

---

Target ML Service
GNIME
Target ML Service
Inversion rosistand Loss
fxc
fNc
JINV
Ideatity
Phase I.
Training fNc and fINV
Phase II. Deploying the ML service
Figure 2: Overview the GNIME pipeline.
convolutional layer before the fully-connected (FC) layers. The activation maps then aggregated by globally pooling all existing channels, thus constituting heatmap, which can be overlaid upon the original image. The layer-wise relevance propagation (LRP) takes holistic approach to explaining model's 's decision B It performs importance decomposition by redistributing relevance scores along the backward pass and derives the relative contribution of each pixel of the imput image.
Model Inversion Attacks. An MI attack refers to an attempt to invert a given model's output, thus computing its approximate imput. The first MI attack targeted simple lincar models to reconstruct certain genetic markers from medical dosage predictions [10]. Fredrikson el al. [9 expanded this idea to target other types of machine learning (ML) algorithms, such as decision trees and shallow neural networks, which enable the attack to target facial recognition models. These studies leveraged the maximum posteriori principle to construct input features that maximize the likelihood of given model's s response. However, their reconstructed images were blurry and of low-resolution; these methods are unfit for reconstructing input images against complicated DNN models.
Yang et al. [29] proposed the idea of utilizing a dedicated DNN model to perform MI attacks. They pretrained this inversion model with imputs that the attacker generates and their corresponding outputs from a target DNN model. This training- baising-based based MI attack drastically improved the model inversion performance. Follow-up works have demonstrated the of generative adversarial networks (GANs) to improve the inversion performance 12] 5 27 311. Recently, Zhao et al. (32) showed that the additional information leakage from XAI models can be further exploited enhance the effectiveness of the MI attack. While previous GAN- based attacks focus on producing generic images of a target class, the explanation-ave instances, such as face images with hats glasses, or earrings. However, to our knowledge, no defense mechanism has been proposed defend against such explanation-aware MI attacks.
attack by Zhao el al. [32] is able reconstruct features specific test
Explanation-aware Model Inversion Attack
Previous studies have demonstrated the critical threat that generative MI attacks impose. In this section, describe the threat model for conducting state-of-the-ard generative MI attacks against which GNIME is designed to protect model explanations. We then explain the state-of-the-art explanation-awas MI attacks using generative networks along with notations.
3.1 Threat Model
We consider target classifier f providing black-box access whereby adversary can query f to obtain its output ÿ in the form of a prediction vector. Each vector value indicates the predicted probability for the corresponding class. Furthermore, the target service leveraging f also provides a model explanation e (e.g., a Grad, Grad-CAM, or LRP saliency map) that illustrates important features contributing to emitting ý.
We assume a man-in- in--int--mdddee adversary who observes a pair of e and á for an input image inaccessible to the adversary. Thus, the adversary's goal is to reconstruct a high-fidelity input image given e and ÿ. Furthermore, we assume that the adversary is able to obtain an auxiliary dataset Dauz whose underlying data distribution is similar to that of the training data D for the target classifier The adversary leverages this auxiliary dataset to train her inversion networks f.. Since the adversary collect (ý, e) by querying J for each input D Dauz, she exploits both ý and train fa-
Note that our adversary model analogous to that of Zhao al. 332 which constitutes strong black-box adversary model. Also, Yang al. [29] assumes similar man-in-the the-middle adversary

---

who reconstructs imput images from prediction vectors observed from f. Our goal is to protect the model explanations from this strong black-box adversary conducting MI attacks.
3.2 Model Inversion Attacks Using Model Explanations
Zhao et al. [2 proposed a novel blxck-box MI attack (ExpMI) that exploits prediction vectors generated from a target classifier f as well the model explanations for each decision. To reconstruct the imput images, they trained a transposed CNN model fA based on auxiliary data, each of which was prediction vector paired with its model explanation.
Consider a target classifier f that is a function f(z) + (e, .) that emits prediction vector and model explanation (i.e., attribution map). The adversary's goal is to compute (e, 5) -> fi, which takes explanation e and 9 to produce 2, revealing sensitive attributes of z. Figure 3 illustrates a simplified structure of the ExpMI model. To take advantage of structural information activation maps, ExpMI utilizes a U-Net-style bypass connection along with flattened inputs directly appended to the intermediate embedding.
U-Net
The adversary takes advantage of e, which leaks sensitive information pertaining to and drasti- cally improves reconstruction performance of JA- As Figure [ shows, the quality of the reconstructed image with is significantly bet- ter than the MI attack using prediction vectors alone generates.
Transposed
Fallen
Figure 3: DNN architecture of ExpMI models.
Considering that model explanation has become instrumental in identifying the reasons behind the decisions of DNN models in various industry sectors, it becomes inevitable for corporations to provide model explanations. However, providing such information also jeopardizes privacy by revealing sensitive attributes reconstructed images. To this end, we propose GNIME, novel defense framework that learns to inject noise into model explanations, thus minimizing the privacy risk of ExpMI attacks and maximizing the utility of noise-injected model explanations.
Learning to Generate Noise for Model Explanations
Figure depicts the overall workflow of GNIME. Phase trains the noise injector and MI attack networks, thus inducing the noise injector network inject inversion-resistae perturbations into model explanations. Phase II then deploys the noise injector network in injecting noise into each model explanation upon its release.
Given a target classifier fxx + (e, û), GNIME learns to inject noise into e to minimize the MI threat. Our approach utilizes the noise generator network (fNG) and inversion network (frNv), both of which are trained upon Dr {x, Mi Ja the original training data of f.
Let SN i) ê be the NG network that generates includes noise. For the INV network, we design fINY (c, ÿ) - â that takes e and ÿ. then computes an inversion image ft. For the loss functions of fNG and fINv, (Eza) to encourage minimum perturbation to the explanations and an image reconstruction loss (Cra) encourage inversion performance. We the pixel-wise mean squared error (MSE) between and è for Lzd, and the MSE between x and à for Lre-
explanation attribution map é, which
explanation distortion loss
Inspired by Generative Adversarial Networks [1I, we form two-player minimax game in which fNG and fINv are trained together using two different objectives. Specifically, the training phase consists of two steps for each epoch.
In the first step, we freeze fNG and train fINY by minimizing the following loss:
(1) The objective improve the inversion capability of fINV given (e, 9) and (é, g). Note that LINY includes the reconstruction loss from the clean explanations in order to prevent the network from overfitting to the perturbed explanations. For the second step, we freeze fINV and train JNG by minimizing the following loss:
LINV (x. (c, + Lre(x, fiNv f)), where fNG(e, 0).
LNG Ln) fINv
(2)

---

Algorithm Training algorithm in Phase I
Input:
target classifier and input data X [nn
noise generator network fNG Generate Y and E from each input X to for number of epochs do
Output:
Sample batch of size m: [(11, 61), (am, Umn em)), consisting of pairs from X, Y, and E Let é [xa (.) in
Update fINV:
VONNN (ere fINv 0:)) Cre fINV (ês, 0:)1
Update fNG:
VeSNG J CCalleeee 8.) (Zre fINV 9:))
It aims to minimize the distortion between and é, while maximizing the error between z and i. Therefore, we train SNG to generate noise-injected attribution map that is similar to the original attribution map but causes JINV to generate an inversion image dissimilar from its original image. Algorithm D Txxplans our training process. In LNG. we set 500 for CelebA models and 100 for MNIST, 10, and ImageNet- 100 models, then deploy the final model after 500 epochs.
In the worst case, fNG will merely cause a regularization effect on fINV, rendering it robust to any noisy explanation. However, as fNG is trained over epochs, it learns to capture and suppress features inside explanations that contain exploitable information regarding the original input image. Once the training is finished, we discard fINv and only make use of fNG for Phase II. For every attribution map e, we use fNG generate a noise-injected attribution map è, then release instead of upon given query z.
Additionally, GNIME service providers inject less noise into the explanations with an arbitrary MSE bound, 7- In this case, GNIME computes a new attribution map e' (see Equation 3) we normalize the noise difference n between é and by dividing by the square root of the variance of over and then add this normalized difference to
e' =
Vol(n)/7 where n = c.
(n) denotes the pixel variance of n. Therefore, by controlling y, GNIME service providers can utilize the trade-off and choose to preserve better explanation functionality in exchange for defensive capability. Refer to the supplementary material for further analysis regarding the trade-off.
5
Experiments
5.1 Experimental Setup
Datasets. We evaluate GNIME using four datasets: (1) CelebA 117. (2) MNIST 14) (3) CIFAR- 10 [13), and (4) ImageNet [6] each of which is freely available for research purposes. For CelebA, use its balanced subset consisting of 30,000 face images of 1000 individuals. To obtain tighter bound for the faces, we crop the center and resize each CelebA image to 128x 128. MNIST consists of 70,000 images of handwritten digits, and CIFAR- 10 has 60,000 images of 10 different objects. Both MNIST and CIFAR- 10 images were resized to 32x32. We note that previous studies [29] 31 32] adopted the same datasets demonstrate how effectively their frameworks mitigate the MI threat. In addition, we evaluate GNIME on ImageNet- 100 [6], which is a large scale dataset of higher resolution, consisting of 100 classes randomly selected classes from ImageNet; each image is resized to 256 x 256.
We split each dataset into two disjoint sets of the same size: the target and attack datasets. The target dataset is used to train a target model f and GNIME. The attack dataset is used to train the inversion networks for explanation-awarc MI attacks and to test GNIME Specifically, we split this attack dataset with an 80/20 ratio for the train/test split.
Note that we assume an adversary capable of training fA upon a dataset of which the underlying distribution is nearly identical to that of the target network's training data. We intentionally assume this strong adversary to demonstrate how good GNIME is in mitigating the MI threat. Moreover, we also assess the performance of GNIME against a weaker but more practical adversary using an auxiliary dataset that has a different underlying distribution (i.c., separately collected set for
5

---

conducting a similar task) (29]. For instance, when attacking the CelebA model, we randomly sample face images from FaceScrub (20] to train fA. Note that the overlapping identities from FaceScrub which also appear in CelebA are removed before the sampling process. We extract tighter bound of each FaceScrub face according to its official bounding box information.
Target Models. We implement different target models for different datasets. For CelebA, the target model consists of three convolutional layers followed by two fully-connected (FC) layers. We employ dropout before each FC layer to avoid overfitting. For MNIST and CIFAR-10, 10, we use a target model with two convolutional layers followed by two FC layers. We selected the same models that Zhao et al. B2 used to demonstrate their MI attacks. For ImageNet-100, we used the model from CelebA with an additional convolutional layer.
Inversion Methods. We use two MI attack methods (ExpMI and PredMI) as baselines in our experiments. Specifically, compare the efficacy of various defense mechanisms against the state- of-the-art ExpMI attack (32) In addition, we consider the prediction-only MI attack using no model explanations (PredMI), suggested by Yang et al. [29]. Note that the inversion results via PredMI form an upper bound that represents the ideal case for any defenses in which model explanations leak no additional knowledge that the adversary is able exploit. We conducted each MI attack five times and reported average metrics with standard deviations. All experiments took place on a system equipped with 512GBs of RAM, two Intel Xeon Gold 6258R CPUs, and four RTX 3090 GPUs.
5.2 Evaluation Metrics
We assess whether the reconstructed image is close to its original image by leveraging four metries: MSE, SSIM, TCA, and DeePSiM, which have been extensively used for quantitative evaluations in previous studies 29 32.
Mean Squared Error (MSE). The MSE is metric that measures the distance between a recon- structed image and its corresponding original image. A small MSE indicates that the reconstructed image is close to its original image. For normalization, input pixel values of the inputs are scaled to be within the range of [0, 1].
Structural Similarity Index Measure (SSIM). The SSIM computes the distortion of structural information between two images (28). Since humans are good at recognizing structural differences between two images, this metric shows human perceptual similarity between two images. An SSIM score spans from 0 to 1, and the higher score indicates better reconstruction.
Deep Perceptual Similarity Metric (DeePSiM). The DeePSiM shows the similarity between two images an intermediate representation level 77). Unlike SSIM, which illustrates perceptual similarity, DeePSiM explains perceptual similarity from the neural network perspective. Specifically, we trained a separate evaluation classifier fE, which shares the same architecture and training dataset with the target model. We then computed the feature distance between intermediate results from the penultimate layer of fE, comparing the high-level features identified by fE (32) fINV accurately reconstructs the high-level image features, it results in high DeePSiM score.
Transferable Classification Accuracy (TCA). We also measure TCA to evaluate whether inverted images are generally recognizable by another DNN classifier of which the structure is the same as that of the target model. This metric indirectly shows whether the reconstructed image incorporates the key features of its original image. We measured this metric using fg. Note that f would correctly predict the reconstructed image if the reconstructed image successfully incorporates the key features of the original image, rather than to overfit to unimportant details captured only by the target classifier.
5.3 Experiment Results
5.3.1 Defending against MI Attacks
We evaluate the performance of vanilla explanation-awar MI attacks and then measure the efficacy of GNIME in decreasing information leakage from such MI attacks. Specifically, we compare five settings (two baseline attacks and three defenses against ExpMI attacks): (1) MI attacks using the model's prediction outputs alone (PredMI): (2) MI attacks using both prediction outputs and model explanations (ExpMT); (3) ExpMI attacks against a defense that injects Gaussian noise into saliency maps (RND); (4) ExpMI attacks against defense that injects optimized noise into saliency maps

---

Table I: Quantitative comparison between five different evaluation settings: PredMI, ExpMI, RND, OND, and GNIME. Values inside brackets indicate the degradation in inversion performance com- pared to ExpMI. The best results in mitigating ExpMI are highlighted in bold.
Grad
Grad-CAM
LRP
Metric
PredMI
ExpMI
RND
OND
GNIME
ExpMI
RND
OND
GNIME
ExpMI
RND
OND
GNIME
50015
20011
20000
2,0002
2.2005
2.0011
5,00
20000
5:000
20035
20002
20011
2:00
MSET
0287
0141
0184
0185
.0222
0183
0191
0280
.229
0060
0124
0155
.0344
(00131)
(00151)
(.00111)
(00091)
(00171)
(0046))
(00561)
(00861)
(01651)
S5IML
4427
6304
5430
.357
3093
5823
3417
5274
.952
8233
6370
5805
.960
(10744)
(11477))
(14101)
(D806])
(0550])
(0871L)
(1863
(24288))
(3273L)
+00000
5.0142
-0571
+0014
2002
+
-000
-0004
-0004
+0033
-0057
TCAL
0217
.000
0690
.8570
.0244
.1097
0860
0795
.4455
.1477
.1359
1207
J8326
(0310))
(D430L)
(07766)
(02371))
(0502])
(0651))
(U588
(
(19501)
(1821L)
+036
-0007
DeePSIM
1851
2420
2117
2064
.1880
.2455
2490
(0053))
(00590)
(
(070001)
(1349))
(
MSE
0237
20001
+000
0067
+000
3070
(0000
1020
2000
0059
806
8177
-000
2012
-0001
500
20006
(00161)
L003ST)
(01561)
L00251)
L015611
MNST
SSIML
50008
.5542
e:0135
5065
8707
8553
3886
2006
9613
9009
20150
-000
9767
9314
(0058))
LOGOS
(100
TCAL
+.0044
9751
2:000
3850
9935
-005
3520
200L
9871
20000
9825
+0200
9715
D867
20004
9835
20002
(0015
(
(1000)))
(0156
(D0S3)
(TESIU)
(11411)
DeePSIM
2:0000
8541
2:0005
9367
41B6
939
2001
20007
9645
2000
20004
2000
3850
DSUS
3874
2017
MSET
200002
5:0000
20005
8416 1416
2000
DET4
2030
DIST
(O0427)
(D0021)
LX0GIT)
(O131T)
EOITOOI
CIFAR-10
SSIML
2.0002
1668
2:0135
.2044
2480
20140
20102
223A
17866
2209
.3348
2,0922
2654
2454
2000
J8SS 8
S007
8862
20014
C
(0475
(
(D6941))
(1894
(0811)
(22934))
2.0058
8:0276
20219
20184
2.040
L0000
20076
20122
TCAL
(05111))
(
(JN67
.952
(
(
29%8
T
2:0001
2:2016
20005
2000
2200
2.0007
1,000
2001
22002
DoePSIM
8344
8500
(0099))
8441
(0059L)
1
(0133
3364
8603
[OPPR])
850S
(DUTS
8485
(02T7L))
X386 8
9378
(0265))
9112
(
5073
MSET
20000
0436
2:0008
0024
0377
0364
J418
.366
D401
LOMI
0377
B425
0166
0361
0309
0006
ImageNNe-11n 200
(00537)
(0000T)
(00944))
(0054))
(OOLIT)
LO0591)
(D0957)
(DIEST)
2.0016
2:0006
10050
2,0019
2.0014
1
200018
1.0015
S5IML
3393
3877
3471
3463
3385
3469
3410
3410
3335
6541
4256
3560
3371
(1405])
(D414L)
(1492))
(009)))
(0134L)
(2285
(2981L)
(31700))
TCA.
0073
.1242
0893
0711
A445
.0615
0514
0469
8388
2669
2067
0876
JM16
((1491))
(D531L)
(11997))
[D101_]
(D145.)
(.0227.))
(06224)
(1813J)
(22778))
2/0004
5/2073
0006
1055
2009
+/201
+00000
2001
-000
5-0000
5.0033
-000
DeePSIM_
3451
3826
5755
5556
3484
3593
3541
3505
3464
.6826
.6490
.333
.5077
(00224)
(0270L)
(.0342))
(0052[])
(008S_]
(00199.)
[0396]]]
(19958L)
(.1319L)
(OND); and (5) ExpMI attacks against GNIME. When measuring the defensive capability of RND, OND, and GNIME, each MI network was separately trained using the explanations perturbed by the corresponding defense method. This resembles the real-world adversary in our black-box setting, who has no way of obtaining clean model explanations.
Table compares the performance of GNIME against these experimental settings assuming a target ML service providing Grad, Grad-CAM, and LRP, respectively. Recall from $3 that we assume a strong adversary in possession of auxiliary data of which the distribution is the same as the training dataset of the target model. We thus use the train split of our attack dataset for training GNIME and the test split for measuring the metrics (5.I). Considering that ExpMI shows better reconstruction quality compared to PredMI in all the cases, exploiting model explanations to perform MI attacks indeed contributes to additional information leakage. Surprisingly, the SSIM score significantly boosted from 0.1688 to 0.8862 when targeting CIFAR-10-LRP. These results demonstrate the necessity of defense against ExpMI attacks. We now compare the performance of GNIME against other two defense settings (RND and OND) that add noise to achieve the same defensive goal. RND corresponds to a naive approach that randomly adds Gaussian noise, which serves as our defense baseline. For OND, instead of training fNG. we use the projected gradient descent [18] to optimize the input image towards increasing the loss function of fINV. Note that adding huge noise can prevent information leakage, but it would undermine the ability to explain the model's decisions. Hence, for fair comparison, we clip the perturbation magnitude of RND and OND to align with the average perturbation size of GNIME.
7

---

B6113
EXMIT
D5409
CATUE
02422
DATTZ
GUOOT
GAOCT
GEC00
BOCE
CARTO
0.0032
0.507
03122
0.0006
2.5004
(a) ExpMI against Grad
(b) ExpMI against Grad-CAM
(c) ExpMI against LRP
Figure 4: For each explanation type, we compare the ExpMI inversion results of four cases: no defense, RND, OND, and GNIME (columns 2-5). The SSIM score of each reconstructed image compared to the target image (column 1) is marked in white.
As shown in the table, GNIME succeeds in decreasing the information leakage threat for all the explanations and data types. Although RND also slightly alleviates the MI threat by degrading the inversion quality, GNIME significantly outperforms RND in all the cases. Furthermore, GNIME shows superior performance compared to OND as well in all cases. For example, when targeting CelebA-LRP, RND and OND lower the TCA by 0.0588 and 0.0940, respectively. On the other hand, GNIME decreases the TCA by 0.1821, which corresponds to 3.10x of RND and 1.94 of OND. From these observations, we conclude that noise optimized against the fINV fails to generalize on fA. which is independently trained for ExpMI attacks, and it requires dedicated model fNG to generate effective mnversion-resistee noise.
PredMI serves as an upper bound for all the presented defense frameworks, which shows the case when there exists additional information leakage threat that stems from providing model explanations (55.I). Considering this upper bound shown in the table, we believe that the degree of GNIME achieving its defensive goal is close to its upper bound. For instance, the DeePSiM score against PredMI and ExpMI targeting CelebA-Grad with GNIME achieves O. 1851 and O. 081880, respectively.
Figure presents the qualitative experimental results. We observe that GNIME outperforms both RND and OND in decreasing the reconstruction quality. When comparing all columns between the third and fifth for each explanation type, GNIME clearly contributes to degrading the quality of reconstructed images; ExpMI still reconstructs most features in the inversion images when the explanations are under the protection of RND or OND, contrary to the inversion images with GNIME.
We further compare the inversion results with a more practical adversary. Here, the adversary cannot train its inversion network fa with the same dataset as the target model (CelebA). Instead, it draws the training data from another generic distribution. Specifically, we use a subset of FaceScrub to train fA- This auxiliary training set is arguably easier to obtain than the original dataset [29].
Table summariesss the ExpMI attack performance with and without GNIME. The metrics consistently report lower reconstruction performance even when the adversary leverages FaceScrub as the auxiliary dataset. Interestingly, the defensive capability of GNIME was even greater in the practical setting (e.g., using FaceScrub to attack CelebA-LRP, the degradation of TCA due to GNIME increased from 84.82% degradation (0.2147-30.0326) 99, 99.43% (..00066000012)) These results demonstrate that GNIME is more effective against the practical adversary of conducting ExpMI attacks.
5.3.2 Preserving XAI Functionality
In this section, we further evaluate whether an attribution map noised by GNIME still provides interpretable explanations on the model's decisions, thus preserving its functionality. The XAI methods (e.g., Grad, Grad-CAM, and LRP) produce an attribution map that reflects the degree of importance for each pixel contributing to a target model emitting decision. Therefore, an input image multiplied by this model explanation (r O e) erodes unimportant pixels and leaves important pixels intact. That is, we can indirectly evaluate whether the original functionality of a model explanation preserved its perturbed version è by comparing the pixels left @ e and x O é using DeePSiM. We compare the intermediate representation values in the penultimate layer of the target classifier
8

---

Table 2: MI attack performance an adversary exploiting auxiliary dataset (FaceScrub) of which distribution is differ- ent from that of the original training set (CelebA).
Table 3: DeePSiM score between @ and ê. Values within paren- theses indicate the average perturba- tion magnitude (in MSE).
Grad
Grad-CAM
LRP
Metric
PreMM
ExphI1
GNIME
ExpMI
GNIME
ExpMI
GNIME
Dataset
Grad
Grad-
LRP
MSE
+1811
.019
+.0010
.0199
(0255 (.086)))
10015
.0212
+0317
0252 (00401)
1000
00318
.0088
(2666
CokbA
3107
.7836
6734
SSIM
+0017
.4257
(179611795))
+0530
(12666)
-0351
-00%
(.375L)
MNIST
9571
.9516
9296
TCAL
=006
0171
=.06dd
1968
(008 (19900))
.0144
(1005))
10004
.2086
(.170L
=0005
CIFAR-
LO126]
(0133)
DePSSM]
20064
=0005
.1755 (.0525))
2011
+0190
1951 (.0667.)
20330
20886
(,341)))
2,001
ImageNett-
(3061)
(00100
9290
for each input: @ and x @ é. A higher DeePSiM score indicates a higher resemblance in the high-level features captured by the target model.
Table shows the average DeePSiM value of all x @ and @ è pairs in which each is from the test split of the attack dataset. The model perceptual similarity between 0.6734, 0.9298, 0,8761, and 0.9097 for the CelebA, MNIST, CIFAR-10 10, and ImageNet- 100 models, respectively. This indicates that the original functionality of e largely remains intact.
e and æ @ is least
Perturbation Magnitude. We analyze the dis- tribution of MSEs between and é pairs for CelebA depicted in Figure The figure shows two triples (z, e, é), each of which respectively represents a common case and an extreme case of perturbation magnitude (i.e., MSE 0.0015 and MSE = 0.0055). Note that even the per- turbation of the extreme case hardly percep- tible. These results suggest that GNIME ap- plies infinitesimal perturbations to such that preserves its original functionality. We include further visualization of perturbed model expla- nations in the supplementary material.
Figure 5: Distribution of GNIME's perturbation magnitude CelebA-Grat explanations. Values marked white inside indicate the perturbation magnitude (in MSE).
In addition, recall from $4 that one can further vary y in Equation to control the perturbation magnitude. We observed linear trade-off relationship between the defensive capability and the explanation functionality of GNIME. GNIME service providers can use y of their choice with this in mind. When choosing to use 0,0040 for y, GNIME achieves the 0.6340 DeePSiM. However, when limiting the perturbation size to 0.0005, the DeePSiM score increases 0.8497. Please refer the supplementary material for further experimental results.
5.3.3 Investigating Additional Information Leakage.
Although GNIME significantly lowers the risk of information leakage to the degree to that of PredMI, information leakage still remains to some extent even after its deployment. However, we stress that GNIME focuses on preventing additional information leakage that stems from model explanations; therefore, the inversion results PredMI form an upper bound of our defense. Note that GNIME is capable of obscuring several key attributes in the original images, providing inversion results similar to those of PredMI. For instance, Figure shows that GNIME removes critical features in the reconstructed images, such sunglasses, cap, gender information.
We additionally evaluate whether GNIME is able obscure several key attributes of reconstructed images. Similar to Chen et al. [3], we trained attribute classifiers using CelebA and evaluated whether the attribute classifiers is able to correctly predict attributes when the reconstructed images are provided. Specifically, we trained eight attribute classifiers, employing the ResNet50V2 architecture from He el al. [12 with its final layer replaced for binary classification. Note that the training set used to train these classifiers is disjoint from that we used in the input reconstruction. We report the Fl score to measure the performance of these attribute classifiers. [ the evaluation results. GNIME performed best in terms of deteriorating the attribute prediction performance for all

---

0.75
11:30or
0.50
D4D
GANN
Carga
Dlack
Canigg
Goldes
Libilles
Mouth Open
Figure 6: Fl scores measured on eight attribute classifiers using reconstructed images.
attribute types. On average, ExpMI without defense reports an Fl score of 0.7672. Note that GNIME decreases 0.3048, which is close the Fl score of PredMI.
6 Discussions and Limitations
Constrained Obfuscation. The concept of obfuscating sensitive information while retaining useful information has been explored in various fields in establishing privacy 4 [6] and fairnsss (30]. Their common approach is to obfuscate given input instances and then jointly train target model to retain a high performance even for those obfuscated instances. By contrast, GNIME does not directly alter the target model parameters and focuses only on obfuscating model explanations to minimize the MI threat. This enables the deployment of legacy models without any retraining steps while preserving original model performance.
Model Extraction. Zhao et al. [32) proposed an ExpMI attack even when a target DNN model does not provide any explanation (ExpMI-3), which we do not assume in the threat model. The core idea of ExpMI-3 is to build a high-performing surrogate model through a successful model extraction (ME) attack and then construct surrogate explanations based on this surrogate model. However, we stress that once the adversary successfully constructs the surrogate model, the surrogate model exposes the target model to numerous white-box attacks, not only limited to the ExpMI threat. We argue that defense against ExpMI-3 should focus on mitigating the ME threat. We implemented simple ME defense, reverse sigmoid perturbation (RSP) 133, and evaluated its efficacy terms of degrading the inversion performance of ExpMI-3; the SSIM score of ExpMI-3 attack against RSP-equipped CelebA was 0.4928, which comparable to the SSIM of ExpMI attack against GNIME-equipped CelebA-LRP, 0.4960. Further experimental results are given in the supplementary material.
Black-box Adversary. Previous MI attacks (5] 31 have been proposed assuming white-box adversary who can access the parameters and gradients of a target model. By contrast, we assume the black-box adversary conducting ExpMI attacks, following Zhao et al. (32. Therefore, GNIME is designed to mitigate the ExpMI threat in a black-box manner in which target model returns its explanation and output upon a given input. Considering that many Maacine--Larring -aagg-as--- -ervice (MLaaS) services operate in black-bor box setting, we emphasize that addressing the ExpMI threat in this setting still important. To our knowledge, GNIME is the first defense approach to effectively mitigate the ExpMI threat.
Conclusion
In this paper, we propose GNIME, the first defense framework against MI attacks using model expla- nations. During the training of GNIME, we put together the noise generator and inversion networks that they compete with each other; the noise generator network learns to inject imperceptible noise into model explanations that undermine model inversion, and the inversion network learns to excel at reconstructing the original inputs from model explanations. GNIME then leverages the trained noise generator network for every model explanation upon its release. Our evaluations across diverse datasets and explanation types demonstrate that GNIME is effective in mitigating the privacy threat that MI attacks impose while preserving the original functionality of model explanations.
Acknowledgementts. This work was supported by the Institute of Information & communications Technology Planning & evaluation (IITP) grant funded by the Korea government (MSIT) 11-222200000155 Penetration Security Testing of ML Model Vulnerabilities and Defense).
10

---

References
Ulrich Aivodji, Alexandre Bolot, and Séhastien Gambs, Model extraction from counterfactual explanations. CoRR, abs/2009.01 000844 2020.
[2] Shengwei An, Guanhong Tao, Qiuling Xu, Yingqi Liu, Guangyu Shen, Yuan Yao, Jingwei Xu, and Xiangyu Zhang. MIRROR: Model inversion for deep learning network with high fidelity, In Proceedings of the Network and Distributed System Security Symposium, 2022.
[3] Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Alaus-Robert Müller, and Wojciech Samek. On pixel-wise explanations for non-linea classifier decisions by layer-wiss wise relevance propagation. PLoS ONE, 10(7), 2015.
[4] Martin Bertran, Natalia Martinez, Afroditi Papadaki, Qiang Qiu, Miguel Rodrigues, Galen Reeves, and Guillermo Sapiro. Adversarially learned representations for information obfuscation and inference. In Proceedings of the International Conference Machine Learning, pages 614-623, 2019.
[5] Si Chen, Mostafa Kahla, Ruoxi Jía, and Guo-Jun Qi. Knowledge-enricced distributional model inversion attacks. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 16178- 16187, 2021.
[6] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 248 208,-255,222 2009.
[7] Alexey Dosovitskiy and Thomas Brox. Generating images with perceptual similarity metrics based on deep networks. In Proceedings of the Advances in Neural Information Processing Systems, pages 2016.
[8] Shihong Fang and Anna Choromanska. Backdoor attacks the dnn interpretation system. In Proceedings of the AAAI Conference Artificial Intelligence, 2022.
[9] Matt Fredrikson. Somesh Jha, and Thomas Ristenpart. Model inversion attacks that exploit confidence information and basic countermeasures. In Proceedings of the ACM Conference on Computer and Communications Security, pages 1322-1333, 2015.
[10] Matthew Fredrikson, Eric Lantz, Somesh Jha. Simon Lin. David Page, and Thomas Ristenpart. Privacy in Pharnaacogenetics: An end-to-end case study of personalized warfarin dosing. In Proceedings of the USENIX Security Symposium, pages 17-32, 2014.
[11] Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David FardeyFarrly, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Proceedings of the Advances in Neural Information Processing Systems, 2014.
[12] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In Proceedings of the European Conference on Computer Vision, pages 1330-44,,7 2016.
[13] Alex Krizhevsky. Learning multiple layers of features from tiny images. Master's thesis, University of Toronto, 2009.
[14] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. In Proceedings of the IEFE, pages 2278-2322 1998.
[15] Taesung Lee, Benjamin Edwards, Ian Molloy, and Dong Su. Defending against neural network model stealing attacks using deceptive perturbations. In JEEE Security and Privacy Workshops, pages 43-49, 2019.
[16] Ang Li, Jiayi Guo, Huanrui Yang, Flora D Salim, and Yiran Chen. DeepObfuscator: Obfuscating intermediate representations with preaeerii adversarial learning on smartphones. In Proceedings of the International Conference Internet-of-Thing Design and Implementation, pages 28-39, 2021.
[17] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3730-3738, 2015.
[18] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks, In Proceedings of the International Conference on Learning Representations, 2018

---

[19] Smitha Milli, Ludwig Schmidt, Anca D. Dragan, and Moritz Hardt. Model reconstruction from model explanations. In Proceedings of the Conference Faimees, Accountability, and Transparency, pages 1-9, 2019.
[20] Hong- Wei Ng and Stefan Winkler. data-driven approach to cleaning large face datasets. In Proceedings of the IEEE International Conference on Image Processing, pages 343-347, 2014.
[21] Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Frad-CAM: Visual explanations from deep networks via gradient-basee localization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 618 626, 2017.
[22] Reza Shokri, Martin Strobel, and Yair Zick. the privacy risks model explanations. Proceedings of the AAAI/ACM Conference on Al, Ethics, and Society, pages 231-241, 2021.
[23] Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagat- ing activation differences. In Proceedings of the International Conference on Machine Learning, pages 3145-3155 2017.
[24] Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. In Proceedings of the International Conference on Learning Representations Workshop, 2013.
[25] Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas. and Martin Wattenberg. Smoothgrad: removing noise by adding noise. CORR, abs/1706 706203225 2017.
[26] Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In Proceedings of the International Conference Machine Learning, pages 3319-3322 2017.
[27] Kuan-Chiel Chinh Wang, Yan Fu, Ke Li, Ashish Khisti, Richard Zemel, and Alireza Makhzani. Variational model inversion attacks, In Proceedings of the Advances in Neural Information Processing Systems, pages 9706 9719, 2021.
[28] Zhou Wang. A.C. Bovik, H.R. Sheikh, and E.P. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Procexsing, 13(4);600- 612, 2004.
[29] Ziqi Yang, Jiyi Zhang. Ec Ec-Chien Chang, and Zhenkai Liang. Neural network inversion in adversarial setting via background knowledge alignment. Proceedings of the ACM Conference on Computer and Communications Security, pages 225 240, 2019.
[30] Rich Zemel, Yu Wu, Kevin Swersky, Toni Pitassi, and Cynthia Dwork. Learning fair representations. In Proceedings of the International Conference on Machine Learning, pages $25-333. 2013.
[31] Yuheng Zhang, Ruoxi Jia, Hengzhi Pei. Wenxiao Wang. Bo Li, and Dawn Song. The secret revealer: Generative model-inversion attacks against deep neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 253-261, 2020.
[32] Xuejun Zhao, Wencan Zhang, Xiaokui Xiao, and Brian Lim. Exploiting explanations for model inversion attacks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 682-692, 2021.
[33] Bolci Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 2921-2929, 2016.
12

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contribu- tions and scope? [Yes], we summarized our contributions the abstract and introduction.
(b) Did you describe the limitations your work? [Yes], we specified the limitations Section
(c) Did you discuss any potential negative societal impacts of your work? [N/A], , we designed and implemented defense framework mitigate MI attacks, which pose no threats or negative impacts to the public.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes] we follow the ethics guideline.
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)? (es), we will submit our artifact in the supplemental materials and PDF file that describes additional experimental results.
(b) Did you specify all the training details (e.g.. data splits, hyperparameters. how they were chosen)? [Yes]. Section 5.1 More details will be provided in the supplementary material.
(c) Did you report error bars (e.g.. with respect to the random seed after running experiments multiple times)? [Yes], we conducted each MI attack five times and reported average metrics with standard deviations. Please reference Tables and 2.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes], We descrihed the computation resources in Section 5.1
We also describe further specifications in the supplementary material.
4. If you using existing assets (e.g., code, data, models) curating/relcasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]. we used (1) CelebA 177 (2)
MNIST [14], (3) CIFAR- 10 [3] (4) FaceScrub 20), and ImageNet 6 to evaluate GNIME, We cited all the datasets in the paper.
(b) Did you mention the license of the assets? [Yss), we mentioned that all datasets are available for research in [..l
(c) Did you include any assets either in the supplemental material URL? [No], we do relase any new datasct.
(d) Did you discuss whether and how consent was obtained from people whose data you'l 're us- ing/curating? [N/A]
(e) Did you discuss whether the data you using/curating contains personally identifiable informa- tion or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "a2jUTNf0WjZ", "reviewer": "Reviewer_aV5M", "summary": "This work proposes a defense method that add noise to model explanations to minimize the risk of model inversion attacks, and preserving the interpretabilities of the explanations. They formulate the problem of defense training as a two-player minimax game between the MI attacker and the noise generator.  The MI attacker takes in the explanation and prediction vector, generates the original input, and the noise generator takes in the model explanation and prediction vector and generates a perturbed version of the explanation in order to prevent the MI attacker from leaking information. The proposed framework is able to defense against MI attack according to the results, achieving a larger reconstruction error than other baselines, and it is able to preserve the effectiveness of the model explanations to some extent. ", "questions": "From the results of the paper, the proposed method achieves better performance than the mentioned baselines. However, the reconstructed image can still reveal some features of the original image. Is there a potential risk that even the revealed partial information may be abused by attackers to reveal privacy of the original image?\n\nThe results provide detailed numbers of performance metrics, but it is not clear whether the improvements are significant or not. Is there a better way to show that the improvements are statistically significant? ", "limitations": "To some extent, but the authors did not mention what other information could be leaked even given the blurry reconstructed images. Ideally, the reconstructed image should be very blurred and cannot be recognized as human face. ", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "originality: the proposal to formulate the defense problem as a min-max game is very interesting and the results show that it works to some extent. The proposed evaluation metrics clearly show the advantage of the proposed method over the baselines such as gaussian noise injection.\n\nquality and clarity: the work's method is clear and experiments include detailed comparisons with several baselines. However, with so many detailed quantitative results, it is not clear whether the achieved results are significant or not. Maybe there could be a better way to present the data so that it is not so messy and could be clearer for readers to understand the results. Plots could be a better option than tables to show the results since they are more straightforward and intuitive. Many experimental details are included in the text, which makes it hard to understand the big picture. Maybe a better way is to have a table summarizing the details of various hyper parameters, and have more subsections each covering one topic. \n\nSignificance: the method itself is interesting and valuable to the community as a way to defend against MI attack. However, it is less clear to me how significant are the results. On one hand, the used dataset is relatively small, what about the results on large scale ImageNet dataset? On the other hand, the reconstructed image even after the noise perturbation on the explanation can still reveal some characteristics of the original image. For example, the gender and some facial features on the human face images can be identified, even if the proposed method achieves better metrics than baselines. \n\n\n\n ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "RZQI-Lya2V-", "reviewer": "Reviewer_FbTU", "summary": "The authors proposed a defense methodology against model inversion attack, which uses model explanation results to recover the training set. The purposes of this defense method are to minimize the attack success rate and keep the model's interpretability. The authors train noise generator and inversion network to achieve these purposes by using a minimax setting. They evaluated the efficacy of their methodology by measuring defense success rate and interpretability using various datasets.\n", "questions": "1. Is there a threat when the adversary knows the fact that the GNIME is used? ", "limitations": "Limitation is enoughly discussed in limitation section of main paper.\n\n<minor suggestion>\n\n1. In line 286, please specify the table.\n2. Please add the attribution maps for qualitative comparison. (Fig 5. is too small to see the difference.)\n\n\n---\nThank you for the additional experiments and explanations.\n\nThe additional experiments show that GNIME is also working for the higher resolution images from Imagenet(256x256).\nHowever, I expected 512 or 1024 resolution of human figures (e.g., FFHQ) or X-ray. \nThese datasets are more important for privacy. \n\nThe explanation for other adversary scenario is great.\nBut I am going to maintain the score since the additional experiments are not resolving weakness.  ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\n1. Proposed first defense method against explanation aware model inversion attack.\n2. Did the same experiment several times to show the consistency of their method.\n\nWeakness\n1. The image data used were limited to low resolution. Since the methodology preserves the privacy of the training set, higher resolution images of human figures or X-rays would be a good choice to check the efficacy of methodology. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "O4pi17GBa34", "reviewer": "Reviewer_KFM9", "summary": "The authors consider the problem of model inversion using an auxiliary dataset identically distributed to the training data, its predictions under the model as well as their explanation maps under Grad, Grad-Cam, and LRP saliency map. Given a trained target classifier, their approach consists in setting up an adversarial game between a noise injection and model inversion networks. The noise injection network returns a censored version of the explanation by minimizing the euclidean distance between the explanation and its censored version while maximizing the euclidean distance between the original image and its reconstruction by the inversion model network. The inversion model network aims to predict the original images from the target model predictions, explanations and the injection network censored explanation.", "questions": "The paper would be substantially strengthened and find a wider audience by considering tabular and text data. \nConsidering large scale, complex image data would give an opportunity for the method to set itself apart even more. \nA theoretical analysis of the algorithm convergence, even in a non-parametric setting and just at optimality, could shed light on the relationship between the noise injection and model inversion network after training. ", "limitations": "Yes.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n* The paper is clearly written and easy to follow.\n* The authors consider a wide range of baselines and quantitative evaluation metric.\n* The author quantitatively show that censored explanations remain interpretable.\n\nWeaknesses:\n* The paper only consider small scale image data. \n* It's not clear whether the algorithm converge or not. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "M1uEOQIsdfk", "reviewer": "Reviewer_GkYa", "summary": "This paper proposes a perturbation-based approach to prevent an adversary from leveraging model explanations to perform model inversion attacks. In particular, the authors apply this defence mechanism to three explanation types: Grad, Grad-CAM, and LRP. The proposed adversarial training framework consists of a model inversion network that performs the inversion attack and a generator network that learn to create perturbations for explanations. Experiments are performed to demonstrate the proposed approach's ability to reduce the efficacy of explanation-aware model inversion attacks.", "questions": "- Please add more discussion on the risk that noisy explanations can lead to manipulated explanations. \n- If possible, report the additional score like the SSIM of $x \\odot e$ and  $x \\odot \\hat{e}$ to provide more arguments for the preservation of XAI functionality.", "limitations": "Nothing to report.", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths** \n\n-   Several evaluation metrics to compare the original input $x$ with the reconstructed input $\\hat{x}$, namely MSE, SSIM, DeePSIM, and TCA\n- The proposed defence mechanism is evaluated against state-of-the-art prediction-based and explanation-aware model inversion attacks\n- Experiments are performed on several real-world datasets\n- Realistic assumptions on the adversary knowledge have been investigated\n\n**Weaknesses** \n\nMy main concern is the lack of discussion on the manipulability of saliency maps. Several research works have shown (e.g., [a]) that these types of explanation techniques can be arbitrarily manipulated. While the paper is not concerned with adversarial manipulation of explanations, the noise added to the original explanations can also unintentionally leads to manipulated explanations that fairwashed the original decisions of the target model. \n\n[a] - Explanations can be manipulated and geometry is to blame. (Dombrowski et al., 2019)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept