Understanding Deep Gradient Leakage via Inversion Influence Functions
Authors: Haobo Zhang
Junyuan Hong
Yuyang Deng
Mehrdad Mahdavi
Jiayu Zhou
Venue: NeurIPS 2023 poster (2023)
Keywords: Deep Learning
Privacy
Federated Learning
Influence Function

Abstract: Deep Gradient Leakage (DGL) is a highly effective attack that recovers private training images from gradient vectors.
This attack casts significant privacy challenges on distributed learning from clients with sensitive data, where clients are required to share gradients.
  Defending against such attacks requires but lacks an understanding of when and how privacy leakage happens, mostly because of the black-box nature of deep networks.
  In this paper, we propose a novel Inversion Influence Function (I$^2$F) that establishes a closed-form connection between the recovered images and the private gradients by implicitly solving the DGL problem.
  Compared to directly solving DGL, I$^2$F is scalable for analyzing deep networks, requiring only oracle access to gradients and Jacobian-vector products.
  We empirically demonstrate that I$^2$F effectively approximated the DGL generally on different model architectures, datasets, modalities, attack implementations, and perturbation-based defenses.
  With this novel tool, we provide insights into effective gradient perturbation directions, the unfairness of privacy protection, and privacy-preferred model initialization.
  Our codes are provided in https://github.com/illidanlab/inversion-influence-function.

Understanding Deep Gradient Leakage via Inversion Influence Functions
Haobo Zhang Michigan State University zhan2060@msu. edu
Junyuan Hong"
Yuyang Deng
Michigan State University University of Texas Austin jyhongdutexas. .dd
Pennsylvania State University yzd820psu. edu
Mehrdad Mahdavi
Jiayu Zhou
Pennsylvania State University mahdavi@cse, psu. psu.edu
Michigan State University
jiayuz@msu.ed
Abstract
Deep Gradient Leakage (DGL) is highly effective attack that recovers private training images from gradient vectors. This attack casts significant privacy chal- lenges distributed learning from clients with sensitive data, where clients required to share gradients. Defending against such attacks requires but lacks understanding of when and how privacy leakage happens, mostly because of the black-box nature of deep networks. In this paper, we propose a novel Inversion Influence Function (I=F) that establishes closed-form connection be- tween the recovered images and the private gradients by implicitly solving the DGL problem. Compared directly solving DGL. I2F is scalable for analyz- ing deep networks, requiring only oracle access to gradients and Jacobian-vector products. We empirically demonstrate that I2F effectively approximated the DGL generally on different model architectures, datasets, modalities, attack imple- mentations, and perturbation- defenses. With this novel tool, we provide insights into effective gradient perturbation directions, the unfairness of privacy protection, and privacy-preferre model initialization. Our codes are provided https: //gs:///tt /gi thub. com haommmll1 wwlllaalaabiinnnnaaaaaaaaaaaaaee inveriieiiio on- inf influence- furctiion
Introduction
With the growing demands for more data and distributed computation resources, distributed learning gains its popularity in training large models from massive and distributed data sources (McMahan et al., 2017; Gemulla et al., 201 I). While most distributed learning requires participating clients to share only gradients or equivalents and provides data confidentiality, adversaries can exploit such information to infer training data. Deep Gradient Leakage (DGL) showed such a real-world risk in distributed learning of deep networks (Zhu et al., 2019; Jin al., 2021). Only given a private gradient and the corresponding model parameters, DGL reverse-engineee the private images with high quality. The astonishing success of the DGL started an intensive race between privacy defenders (Sun et al., 2020; Wei et al., 2021; Gao et al., 2021; Scheliga et al., 2022) and attackers (Geiping al., 2020; Zhao et al., 2020; Jin al., 2021; Jeon et al., 2021: Zhu and Blaschko, 2021; Fowl et al., 2022).
the gradient and recovers
crucial question be answered behind the adversarial game is when and how DGL happens, understanding of which is fundamental for designing attacks and defenses. Recent ad adaances in
Equul contribution.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

theoretical analysis shed light on the question: Balunović et al. (2022) unified different attacks into a Bayesian framework, leading to Bayesian optimal adversaries, showing that finding the optimal solution for the DGL problem might be non-trivial and heavily relies on prior knowledge. Pan et al. (2022); Wang et al. (2023b) advanced the understanding of the leakage in specific class of network architectures, for example, full-connected networks. However, it remains unclear how the privacy leakage happens for a broader range of models, for example, the deep convolutional networks, and there is pressing need for model-agnostic analysis.
In this paper, we answer this question by tracing the recovered sample by DGL algorithms back to the private gradient, where the privacy leakage ultimately derives from. To formalize the optimal DGL adversary without practical difficulties in optimization, we ask the counterpart question: what would happen if we slightly change the private gradient?
Answering this question by perturbing the gradient and re-evaluating the attack can be prohibitive due to the difficulty of converging to the optimal attack in a highly non-convex space. Accordingly, we propose novel Inversion Influence Function (IPF) that provides an analytical description of the connection between gradient inversion and perturbation. Compared to directly solving DGL, the analytical solution in F'F efficiently scales up for deep networks and requires only oracle access to gradients and Jacobian-vecte function (Koh and Liang, 2017; Hampel, 1974) that describes how optimal model parameters would change upon the perturbation of a sample and I2 can be considered as an extension of the influence function from model training to gradient inversion.
products. We note that F shares the same spirit of the influence
The proposed F characterizes leakage through the lens of private gradients and provides a powerful tool and new perspectives inspect the privacy leakage: (1) First, find that gradient perturbation is not homogeneous in protecting the privacy and is more effective if it aligns with the Jacobian singular vector with smaller singular values. (2) As the Jacobian hinges on the samples, the variety of their Jacobian structures may result in unfair privacy protection under homogeneous Gaussian noise. (3) We also examine how the initialization of model parameters reshapes the Jocabian and therefore leads to quite distinct privacy risks.
These new insights provide useful tips on how to defend DGL, such perturbing gradient in specific directions rather than homogeneously, watching the unfairness of protection, and carefully initializing models. We envision such insights could lead to the development of fine-grained privacy protection mechanisms. Overall, our contributions can be summarized as follows. (1) For the first time, we introduce the influence function for analyzing DGL; (2) We show both theoretical and empirical evidence of the effectiveness of the proposed 2FF which efficiently approximates the DGL in different settings; (3) The tool brings in multiple new insights into when and how privacy leakage happens and provides a tool to gauge improved design of attack and defense in the future.
2 Related Work
Attack. Deep Gradient Leakage (DGL) is the first practical privacy attack on deep networks (Zhu et al., 2019) by only matching gradients of private and synthetic samples. As accurately optimizing the objective could be non-trivial for nonlinear networks, a line of work has been developed to strengthen the attack (Zhao et al., 2020). Geiping et al. (2020) introduce scaling invariance via cosine similarity. Balunović et al. (2022) summarizes these attacks as a variety of prior choices. The theorem echoed by empirical studies that show better attacks using advanced image priors (Jeon et al., 2021). It was shown that allowing architecture modification could weigh higher risks in specific cases (Zhu and Blaschko, 2021; Fowl et al., 2022). Besides the study of gradient leakage in initialized models, recent evidence showed that the well-trained deep network can also leak the private samples (Haim et al., 2022). Yet, such an attack still requires strict conditions to succeed, c.g., small datasct and gradient flow assumptions. Though these attacks were shown to be promising, exactly solving the attack problem for all types of models data is still time-consuming and can be intractable in reasonable time (Huang et al., 2021).
Defense. Most defense mechanisms introduce perturbation in the pipeline of gradient computation. 1) Gradient Noise, Since the DGL centered on reverse engineering gradients, the straightforward defense is to modify the gradient itself. Zhu et al. (2019) investigated a simple defense by Gaussian noising gradients or pruning lag-magn parameters-ban -based defense. The Gaussian noise was motivated by Differential Privacy Gradient Descent (Dwork, 2006; Abadi et al., 2016), which

---

guarantees privacy in an aggregated gradient. Except for random perturbation, the perturbation could also be guided by representation sensitivity (Sun al., 2020) or information bottleneck Scheliga et al. (2022). 2) Sample Noise. The intuition of sample noise is to change the source of the private gradient. For example, Gao et al. (2021) proposed to seek transformations such that the transformed images are hard to be recovered. Without extensive searching, simply using strong random data augmentation, e.g., MixUp (Zhang et al., 2017), can also help defense (Huang et al., 2021). If public data are available, hiding private information inside public samples is also an alternative solution (Huang et al., 2020). Though the defenses showed considerate protection against attacks in many empirical studies, the security still conceptually relied the expected risks of non-trivial number of evaluations. Explanation of how well a single-shot protection, for example, noising the gradient, is essential yet remains unclear.
Understanding DGI. There are many recent efforts in understanding when noise-based mechanisms help privacy protection. Though Differential Privacy (DP) provides theoretical guarantee for general privacy, it describes the worst-case protection and does not utilize the behavior of DGL., motivating a line of studies to understand when and how the DGL happens. Sun et al. (2020) connected the privacy leakage to the representation, which aligns with later work on the importance of representation statistics in batch-normalizatioon layers (Hatamizadeh et al., 2021; Huang et al., 2021). Balunović et al. (2022) unified different attacks in a Bayesian framework where existing attacks differ mainly on the prior probability. Chen and Campbell (2021) investigated the model structure to identify the key reasons for gradient leakage. Hatamizadeh et al. (2021); Huang et al. (2021) evaluated the DGL in practical settings and showed that BN statistics is critical for inversion. There are efforts advancing theoretical understandings of DGL.. Pan et al. (2022) studied the security boundary of ReLU activation functions. In specific class of fully-connected networks, Wang et al. (2023b) showed that a single gradient query at randomized model parameters like DP protection can be used to reconstruct the private sample. Hayes et al. (2023); Fan et al. (2020) proposed an upper bound of the success probability of the data reconstruction attack and the relative recovery error, respectively, which is considered as the least privacy risk. Instead, we consider the worst-case privacy risk.
Though many useful insights are provided, existing approaches typically focus on specific model architectures. This work provides new tool for analyzing the DGL without assumptions about the model architectures or attacking optimization. Independent of the model architectures, our main observation is that the Jacobian, the joint derivative of loss w.r.t. input and parameters, is the crux.
3 Inversion Influence Function
In this section, we propose new tool unravel the black box of DGL. We consider general loss function L(x,,0 0) defined on the joint space of data X' and parameter e For simplicity, we assume x is an image with oracle supervision if considering supervised learning. Let zy be the private data and go & [z 8) be its corresponding gradient which is treated as constant. An inversion To. Deep Gradient Leakage (DGL) attacks by directly searching for synthetic sample to reproduce the private gradient g by VeL(x, 0) (Zhu et al., 2019). Formally, the DGL is defined as a mapping from gradient g to a synthetic sample ry:
attack aims at reverse engineering go to obtain
2 = Gg) e arg min (Lr(z;g) & | 0) - =
(1)
#EX
where L is the inversion loss matching two gradients, ||||| denotes the f--norm either for vector or
a matrix. The L2 bo-norm of a matrix A is defined by the induced norm ||||| = sup (I / ||=]]).
The Euclidean distance induced by fn-norm can be replaced by other distances, e.g., cosine similar- ity (Geiping et al., 2020), which is out of our scope, though.
Our goal is to understand how the gradient of a deep network at input zu encloses the information of xp. For a linear model, e.g., L(x, 0) = x T 0, the gradient is x and directly exposes the direction of the private sample. However, the task gets complicated when a deep neural network is used and L(a, 0) is a nonlinear loss function. The complicated functional makes exactly solving the minimization in Eq. (1) highly non-trivial and requires intensive engineering in hyperparameters and computation (Huang et al., 2021). Therefore, analyses tied to a specific DGL algorithm may not be generalizable. We introduce an assumption of a perfect attacker to achieve a generalizable analysis:
Assumption 3.1. Given gradient vector g, there is only a unique minimizer for Ly g).
3

---

Since VoL(z, 6) is minimizer for (z(V)((z,))) 8)), Assumption 3.1 implies that (VoLz, 0)) is a perfect inversion attack that recovers exactly. The assumption of perfect attacker considers the worst-case. It allows us to develop a concrete analysis, even though such an attack may not always be feasible in practice, for example, in non-linear deep networks. To start, we outline the desired properties for our privacy analysis. (1) Efficiency: Privacy evaluation should be efficient in terms of computation and memory; (2) Proximity: The alternative should provide a good approximation or a lower bound of the risk, at least in the high-risk region; (3) Generality: The evaluation should be general for different models, datasets, and attacks.
3.1 Perturbing the Private Gradient
To figure out the association between the leakage and the gradient we formalize a counterfactual: what kind of defense can diminish the leakage? A general noise-based defense can be written as g (xD, 0) + 5 where is small perturbation. According to Assumption 3.1, a zero d is not private. Instead, we are interested in non-zero 6 and the recovery error (RE) of the recovered image:
RE(20, Gr + 6)) - 3 Gr + 6)ll
(2)
For further analysis, we make two common and essential assumptions follows.
Assumption 3.2. L(x, f) is twice-diffent differeiffee w.r.. x and 6, i.e., J & V VoVe((2, 0) E Rd.xds exist, where de and de are the dimensions of and 0 respectively.
Assumption 3.3. JJ is invertible.
We then approximate Ggo + +) by the first-order Taylor expansion:
Gr + 5) R G9(9)) + (Gr( ā xo
8Gr(gn) 5.
+
(3)
dgo
agu
By the implicit function theorem, we can show that aG.(m) (JJJ)-L (proof in Appendix B.1). Ago
Thus, for small perturbation 8, approximate the privacy leakage through DGL by
Inversion Influence Function (1²F): |gG + 6) xoll I(d; - [[ (4)
The I'F includes a matrix inversion, computing which may be expensive and unstable for singular matrixes. Thus, we use tractable lower bound of I2F as:
|| A [mb6;
(JJJ)~
(5)
Ammax (JJT)
where (A) denotes the maximal eigenvalues of matrix A. Computing the maximal eigenvalue usually much cheaper than the matrix inversion. We note that the approximation uses Assumption 3.1 and provides general risk measurement of different attacks.
is
The lower bound describes how risky the gradient is in the worst case. This follows the similar intuition of Differential Privacy (DP) (Dwork, 2006) to bound the max chance (worst case) of identifying private sample from statistic aggregation. The proposed lower bound differentiates the DP worst case in that it leverages the problem structure of gradient inversion and therefore characterizes the DGL risk more precisely.
Efficient Evaluation. The evaluation of I or its lower bound Ipp is non-trivial due to three parts: computation of the Jacobian, the matrix inversion, and eigenvalue. The former two computations can be efficiently evaluated using established techniques, for example, (Koh and Liang, 2017). 1) Efficient evaluation of Jō. Instead of computing the dense Jacobian, an efficient alternative is to leverage the lacobian-vector product, which similar to the Hessian-vectorr product. We can efficiently evaluate the product of Jacobian and 8 by rewriting the product as Jô v L 0)o). Since VJ L(xn, 0)ö is scalar, the second derivative w.r.t. can be efficiently evaluated by autograd tools, e.g., PyTorch. The computation complexity is equivalent to two times of gradient evaluation in addition to one vector production. 2) Efficient matrix inversion. We can compute A (JJJ)-1-8 by solving min; || J TTll whose gradient computation only contains two Jacobian- There are other alternative techniques, for example, Neumann series or stochastic approximation, discussed in (Koh and Liang, 2017). For brevity, discuss these alternatives in Appendix A.
products.

---

In this paper, we use the least square trick and solve the minimization by gradient descent directly. 3) Efficient evaluation of privacy with the norm of Jacobian. Because that || Aumax (JJT), we need to compute the maximum eigenvalue of JJ", which can be done efficiently by power iteration and the trick of Jacobian-vector products.
Complexity. Suppose the computation complexity for evaluating gradient is constant in the order Old) where d is the maximum between the dimension of f and z. Then evaluating the Jacobian- vector product is of complexity ((d). By the least square trick with '" iterations, we can compute I in (dd)) time. The lower bound Zω replaces the computation of matrix inversion with computation on the max eigenvalue. By T step power iteration, the complexity is (d((T + T-
Case study Gaussian perturbation. Gaussian noise is induced from DP-SGD (Abadi et al., 2016) where gradients are noised to avoid privacy leakage. Assuming the perturbation follows a standard Gaussian distribution, we can compute the expectation follows:
E IRRE (zo, Go + +)]] (II
= Σ (J)),
where {J J) is the i-th eigenvalue of J' The above illustrates the critical role of the Jacobian eigenvalues. For a perturbation to offer a good defense and yield a large recovery error, the Jacobian should have at least one small absolute eigenvalue. In other words, the protection is great when the Jacobian is nearly singular or rank deficient. The intuition for the insight is that a large eigenvalue indicates the high sensitivity of gradients the sample, and therefore an inaccurate synthesized sample will be quickly gauged by the increased inversion loss.
3.2 Theoretic Validation
The proposed IF assumes that the perturbation is small enough for accurate approximation, and yet is interesting to know how good the approximation is for larger 8. To study the scenario, we provide a lower bound of the inversion error on a non-infinitesimal 8, using weak Lipschitz assumptions for any two samples x and x'.
Assumption 3.4. There exists PJ > 0 such that I VoL(s, V. VoVoa 0)1 S aJ || x'
Assumption 3.5. There exists PL > 0 such that VoL(x, 0) VeL(I' ,) PL ||x - z'1l.
Theorem 3.1. If Assumption 3.4 and 3.5 hold, then the recovery ervor satisfies:
|
Ii - G9 - + +8)1 2
AL ||||| + 2n 1190 - 81|
where J V VoL(o,
The proof is available in Appendix B.2. Similar to IFF, the inversion error could be lower bound using the Jacobian matrix. Eq. (7) matches our lower bound Ip in Eq. (5) with an extra term of gradients. The bound implies that approximation could be effective as a lower bound even for larger 8. Besides, the lower bound provide additional information when is large: when ||ö]| gets infinitely large, the bound converges to Q([] /(u [])l)), which does not scale up infinity. The intuition is that the x is not necessarily unbounded for matching an unbounded 'gradient'.
4 Empirical Validation and Extensions
We empirically show that our metric is general for different attacks with different models on different datasets from different modalities.
Setup. We evaluate our metric on two image-classffication datasets: MNIST (LeCun, 1998) and CI- FARI0 (Krizhevsky et al., 2009). We use a linear model and a non-convex deep model ResNet18 (He et al., 2015a) (RN18) trained with cross-entropy loss. We evaluate our metric on two popular attacks: the DGL attack (Zhu et al., 2019) and the GS attack (Geiping et al., 2020). DGL attack minimizes the La distance between the synthesized and ground-truth gradient, while the GS attack maximizes the cosine similarity between the synthesized and ground-truth gradient. We use the Adam optimizer with a learning rate of 0. to optimize the dummy data. Per attacking iteration, the dummy data projected into the range of the ground-trute data [0, with the GS attack. When the DGL attack
5

---

(mattrx inversion)
10
(matrrix inversion!
20r
10
10
RMSE
RMSE
10-1
10-1
16-1
10 Za (matrix nom))
306
10-1 15-
2014
Anom (nominator only)
Ip (matrix norm)
Trom (nominator only)
Compare different approximation
Compare F'F RMSE
Figure I: Value comparisons attacking ResNet18 on MNIST by DGL, where the grey line indicates the equal values and darker dots imply smaller Gaussian perturbation 6. (a), the y-axis is calculated as defined in Eq. (4) and Iι is calculated as defined in Eq. (5). I lower bound (In) provides good approximation to the exact value with matrix inversion and the root of mean square enror (RMSE) of recovered images. Instead, removing the denominator in Z results in overestimated risks.
RMSE
AMSE
RMSE
RMSE
10-1
10
16:1
10.7
12-1
327
12:1
Za
Ze
(a) CIFARIO, DGL, RNI8 (b) CIFARIO. RNI8
(c) MNIST, RNI8
(d) MNIST, GS, Linear
Figure 2: I2F works under different settings: datasets, attacks, and models. The grey line indicates the equal values, and darker dots imply smaller Gaussian perturbation 6.
cannot effectively recover the data, e.g., on noised linear gradients, only conduct the GS attack to show the highest risk. We use the root MSE (RMSE) to measure the difference between ground-truth and recovered images. All the experiments are conducted on one NVIDIA RTX A5000 GPU with the PyTorch framework. More details and extended results are attached in Appendix C.
Extension to singular Jacobians. We consider the JJ be singular, which often happens with deep neural networks where many eigenvalues are much smaller than the maximum. this case, the inversion of JJJ does not exist or could result in unbounded outputs. To avoid the numerical
instability, we use [JJJ + cl) -1 - with a constant c. In our experiments, the treated as hyperparameter and will be tuned on samples to best fit the linear relation between RMSE and I.
Batch data. In practice, an attacker may only be able carry out batch gradient inverting, 1 VoT. 0)
Σ. gill. which can be upper bounded by 1 Σ. [[V,L[ 0) 9: ||- Such decomposition can be implemented following (Wen al., 2022), which modifies the parameters of the final layer to reduce the averaged gradient to an update of single sample. Thus, we only consider the more severe privacy leakage by individual samples instead of batch of data.
How well are the influence approximations? In Fig. 1(a), we compare the two approximations of I'F Ins relaxes the matrix inversion in I to the reciprocal of || Inom further drops the denominator and only keep the nominator [|] The two figures show that Th serves tight approximation the real influence while Inom much smaller than the desired value. In Fig. 1(b), we see that the relaxation Ib is also a good approximation lower bound for the RMSE.
Validation on different models, datasets, and attacks. In Fig. 2, we show that I'F can work approximately well in different cases though the approximation accuracy varies by different cases. (I) Attack: Figs. 2(a) and 2(b), we compare DGL to the Gradient Similarly (GS) (Geiping et al., 2020) that matches gradient using cosine similarity instead of the L2 distance. We see that different attacks do not differ significantly.
(2) Dutaset: As shown Figs. 2(b) and 2(c), different datasets could significantly differ regarding attack effectiveness. The relatively high RMSE implies that CIFAR tends to be harder to attack due to more complicated features than MNIST. Our metric suggests that there may exist stronger inversion attack causing even lower RMSE. (3) Model:

---

PF=5x10- Pr=4x10-1 PF=1x10-2
PF=11
0.4
LPIPS=0.24
LPIPS=0.42 LPIPS=0.48 LPIPS=0.66
0.3
10
10th
10
100
(a)
(b)
Figure Evaluation of I'F on Re and ImageNet. (a): Darker color means larger noise variance. LPIPS is used to evaluate the semantic information of the recovered and original images. It is good estimator of the semantic distance between the recovered images and original images. (b): Original (top) and recovered (bottom) images with their corresponding I and LPIPS. Images with lower F also have a smaller LPIPS, which implies a better reconstruction.
Comparing Figs. 2(c) and 2(d), the linear model presents better correlation than the ResNet 8 (RN18). Because the linear model more likely be convex, the first-order Taylor expansion in Eq. (3) can approximate its RMSE than one of the deep networks.
Results on Large Models and Datasets. We also evaluate our I2F metric on ResNet152 with ImageNet. For larger models, the RMSE is no longer a good metric for the recovery evaluation. Even state-of-the-an attacks are used and the recovered image visually similar to the original image in Fig. 3(a), the two images are measured to be different by RMSE, due to the visual shift: The dog head shifted toward the left side. To capture such shifted similarity, we use LPIPS (Zhang et al., 2018) instead, which measures the semantic distance between two images instead of the pixel-to-pixel distance like RMSE. Fig. 3(a) shows that 12F is correlated to LPIPS using large models and image scales. This implies that 12F is good estimator of recovery similarity. In Fig. 3(b), original images with lower 12F also have a smaller LPIPS, which means better reconstruction. Recovered images on the right (the German shepherd and the panda) cannot be recognized while those on the left (the owl and the beagle) still have enough semantic information for humans to recognize.
Results on Language Models and Datasets. Since the input space of language models is not continuous like images, current attacks on images cannot be directly transferred text. Besides, continuously optimized tokens or embeddings should be projected back to the discrete text, which induces a gap between the optimal and the original text. Thus, the investigation of privacy risks in text is an interesting yet non-trivial problem. We evaluate the proposed I2F metric BERT (Devlin al., 2018) and GPT-2 (Radford al., 2019) with TAG attack (Deng et al., 2021), which minimizes Le and L1 distance between gradients of the original and recovered images. We use ROUGE-L (Lin, 2004) and Google BLEU (Wu et al., 2016) to measure the semantic similarity between the original text and the recovered text. More experimental details and results with ROUGE-1 and feature MSE are provided C.2. The results are presented Fig. 4. darker color indicates a larger noise variance. Since the ROUGE-L and Google BLEU measure the semantic similarity of the text pair while our metric estimates the difference, two semantic metrics are negatively correlated to Ig-- The results demonstrate the effectiveness of our metric in evaluating the privacy risks of discrete data.
5 When Does Privacy Leakage Happen?
Guided by the formulation of F'F, we provide insights on when privacy leakage happens based on the proposed metric. Our insights are rooted the properties of Jacobian. Since we want to consider the whole property of the Jacobian matrix, choose a shallow convolutional neural network LeNet (LeCun et al., 1998) to trade off the utility and efficiency experiments by default.
5.1 Perturbations Directions Are Not Equivalent
Implication choosing 8. Eq. (4) clearly implies that the perturbation is not equal in different directions. Decomposing = UEV using Singular Value Decomposition (SVD), we obtain I(5; -V- 'V 8|.. Thus, 5 tends to yield larger I2F value if it aligns with the directions of small eigenvalues of J.J

---

6x10
6x.00
Google BLEU
6x10
Google BLEU
ROUGE-L
ROUGE-L
3x
10-3
3x10
10-1
10-1
10-1
10
10th
10 10
10 10 10
10
10aH
10
10a
10
16 10t2 10
10
To
Tp
Tib
(a) ROUGE-L
(b) Google BLEU
(c) ROUGE-L
(d) Google BLEU
Figure 4: Evaluation of Za on BERT (a-b) and GPT-2 (c-d). A darker color means a larger noise variance. ROUGE-L and Google BLEU are used to evaluate the semantic similarity between the original text and the recovered text. Zg is linearly correlated to the two semantic metries, which means In can be used to estimate the privacy risk of the private text.
0.00
A-02
A-2
1.0
3.5
4
0.8
2.5
0.6
1.5
A-22
0.2
0.5
7
152
10:11
122
12:2
10a
(a) CIFARIO
(b) MNIST
Figure 6: Same perturbation sizes but different protection effects by different eigenvectors. In (a)
and (b), MSEs of DGL attacks are reversely proportional to eigenvalues on the LeNet model. Blue curves are scaled 1 1/A. Darker dots indicate smaller MSE (higher risks). Recovered MNIST images associated with different eigenvalues are present on the right.
CIFAR1O
MNIST
Singular values of Jacobian of deep networks. In Fig. 5, we examine the eigenvalues of LeNet on the MNIST and CIFAR10 datasets. On both datasets, there are always a few eigenvalues of J (around 10-2) that much smaller than the largest one (2 1). Observing such large gap between JJ cigenvalues, it is natural to ask: how do the maximal/minimal eigenvalues affect the DGL attacks?
20-1
10-2
normall zed Index
Comparing eigenvectors in defending DGL. of perturbation by letting be an eigenvector of JJ Then the I'F will be 1/A where À is the corresponding eigenvalue. We conjecture 1/A could predict the MSE of DGL attacks. To verify the conjecture, we choose 4 eigenvectors with distinct eigenvalues per sample. The results for the LeNet model are present in Fig. 6. We that the MSE decreases by X. For the MNIST dataset, the MSE-À relation is very close to the predicted 1/A. Though the curve biased from the ground truth for CIFARIO. 10, still can use 1/A lower bound the recovery error. The bias in CIFARIO is probably due to the hardness recovering the more complicated patterns than the digit images. The recovered images in Fig. 6 suggest that with the same perturbation scale, there exist many bad directions for defense. In the worst case, the image can be fully covered. The observation is an alerting message the community: protection using random noise may leak private information.
We consider a special case
Figure 5: The distribu-
tion of eigenvalues of JJJT of two datasets the LeNet model.
5.2 Privacy Protection Could Be Unfair
Our analysis of the Gaussian perturbation in Eq. (6) indicates that the privacy risk hinges on the Jacobian and therefore the sample. We conjecture that the resulting privacy risk will vary significantly due to the sample dependence. In Fig. 7, we conduct fine ine-grained experiments to study the variance of privacy protection, when Gaussian perturbation on gradients used for protection. We use well-trained model which is thought to be less risky than initial models, for example, in (Balunović et al., 2022). Though the average MSE implies reasonable privacy degree as reported previous literature, the large variance delivers the opposite message that some samples or classes are not that

---

6
35
original
2
4
probabiing
1.
MSE
class
Figure 7: The sample-wise and class-wise statistics of the DGL MSE on the MNIST dataset, when gradients are perturbed with Gaussian noise of variance 10-3. The purple lines indicate the average values. Large variances are observed among samples and classes. The original (top) and recovered (bottom) images for the well- and poorly-protecee classes are depicted on the right side.
safe. In the sense of samples, many samples are more vulnerable than the average. For the classes, some classes are obviously more secure than others. Thus, when the traditional metric focusing on average is used, may deliver a fake sense of protection unfairly for specific classes or samples.
To understand which kind of samples could be more vulnerable to attack, we look into the Jacobian and ask when JJJ will have larger eigenvalues and higher risks in Fig. 7. Consider a poisoning attack, where an attacker aims to maximize the training loss by adding noise 5. to the sample I. Suppose the parameter is updated via 0' (2 55) 0 V òz, 0). Let Js V Vo + 0) and we can derive the maximal loss amplification on a test sample 21 when perturbing æ as:
max Ez, [L 8' (x + 6z))] ||,.1|51
max Ez,, [L(x1, 0 (z))] 116.1151
Ez, 10 (. (x))
max 8 JoE [e L0 116.1111
Ja ( L( 6')l
S I| || (V L
As shown by the inequality, the sample with large || may significantly bias the training after mild perturbation 65. If Assumption 3.4 holds, then ||-]| can be associated with |Js|| by || ||Js|l s
J... Now we can connect the privacy vulnerability to the data poisoning. With Eq. (6), samples that suffer high privacy risks due to large || could also be influential in training and can be easily used to poison training.
5.3 Model Initialization Matters
Since Jocabian leans on the parameters, we hypothesize that the way we initialize a model can also impact privacy. Like previous work (Sun et al., 2020; Balunović et al., 2022; Zhu al., 2019), are interested in the privacy risks when a model is randomly initialized without training. Unlike previous work, further ask which initialization strategy could favor privacy protection under the same Gaussian perturbation on gradients.
Case Study: One-layer network. A
1.6
one-layer network with nonlinear activation at)) is simple yet sufficient model for analyzing initialization. Let L(x,0) 0)
MSE
1.6
||o(O 2) bll2 where b is constant. De- note the product A x as a. The Jacobian is given by:
uniform lallgnng
0.8
kalming
uniform
normal
normal
xavier
xavier
22
107
207
10"
12
16'
=[1²]
ØL do
(a) MNIST
(b) CIFARIO
0r00 Da ((0(0)2)) b)1 =00}
Figure 8: Different initialization strategies could re- (1) Apparently, a well-trained 0 with sult in distinct MSES.
(o(0))) x) b) = will cause J/ to be rank-one matrix. Note that the gradient will be zero in such a case and no longer provide any information about the private sample. Consistent with our theoretical results, where the inverse Jacobian makes I infinite, the gradient will be perfectly private at any perturbation. (2) If (8 I)- - is non non-zero, the singular values of J will depends on how close E[a(( xr)] is If E( x)) approaches b at initialization, then will approximately singular, and therefore the initialization enhances privacy preservation.

---

Comparing initialization. To evaluate the effect of initialization on the inversion, we conduct experiments of the LeNet model on MNIST and CIFARIO datasets with DGL and GS attacks. We add Gaussian noise to the gradients, which implies an expected MSE should be proportional to B > as proved in Eq. (6). Here, we evaluate how the initialization will change eigenvalues of JJ and thus change the MSE of DGL.. Four commonly used initialization techniques are considered here: uniform distribution in range [- 0.5, 0.5), normal distribution with zero mean and variance 0.5, kaiming (He et al., 2015b) and xaviez er (Glorot and Bengio, 2010) initialization. For each initialization, the same 10 samples are used to conduct the attack three times. The results are shown in Fig. 8. We observe significant gap between initialization mechanisms. Using uniform initialization cast serious risks of leaking privacy under the same Gaussian defense. Though not as significant as uniform initialization, the normal initialization is riskier than rest two techniques. kaiming and xavier methods can favor convergence deep learning and here we show that they are also preferred for privacy. A potential reason is that the two methods can better normalize the activations to promote the Jacobian singularity.
In parallel with our work, Wang et al. (2023a) also finds that the initialization is impactful on inversion attacks. Despite their analysis being based on different metric, the layer-wise variance of the model weights, their work and ours have a consistent conclusion that models initialized with a uniform distribution face higher privacy risks than that with kaiming distribution
6 Conclusion and Discussion
In this paper, we introduce a novel way to use the influence functions for analyzing Deep Gradient Leakage (DGL). We propose new and efficient approximation of DGL called the Inversion Influence Function (FF)). By utilizing this tool, we gain valuable insights into the occurrence and mechanisms of DGL, which can greatly help the future development of effective defense methods.
Limitations. Our work may be limited by some assumptions and approximations. First, we worked on the worst-case scenario where a strong attack conducts perfect inversion attacks. In practice, such an assumption can be strong, especially for highly complicated deep networks. The gap between existing attacks and perfect attacks sometimes leads notable bias. However, we note that recent years witnessed many techniques that significantly improved attacking capability (Geiping et al., 2020; Jeon et al., 2021; Zhao al., 2020), and our work is valuable to bound the risks when the attacks get even stronger over time. Second, similar to the traditional influence function, 12F can be less accurate and suffers from large variance in extremely non-convex loss functions. Advanced linearization techniques (Bac et al., 2022) can be helpful in improving the accuracy of influence. Then extending our analysis to bigger foundation models may bring intriguing insights into the scaling law of privacy.
Future Directions. As the first attempt influence function in DGL, our method can serve multiple purposes to benefit future research. For example, our metric can be used to efficiently examine the privacy breach before sending gradients to third parties. Since I'F provides an efficient evaluation of the MSE, it may be directly optimized in conjunction with the loss of main tasks. Such joint optimization could bring in the explicit trade-off between utility and privacy in time. In comparison, traditional arts like differential privacy are complicated by tuning the privacy parameter for the trade-off. Furthermore, analysis. For example, unrolling-basse analysis leverages the iterative derivatives in the DGL to uncover the effectiveness of gradient perturbations (Pruthi et al., 2020).
envision that many techniques
be adopted further enhance the
Broader Impacts. Data privacy has been a long-term challenge in machine learning. Our work provides a fundamental tool to diagnose privacy breaches in the gradients of deep networks. Under- standing when and how privacy leakage happens can essentially help the development of defenses. For example, it can be used for designing stronger attacks, which leads to improved defense mechanisms and ultimately benefit the privacy and security of machine learning.
Acknowledgments
This research was supported by the National Science Foundation (ISS-2112174, IIS-1749940), National Institute of Aging (G0F24447444)),, and the Office of Naval Research (N00014- 20-1- 2382).
10

---

References
Abadi, M., Chu, A., Goodfellow, L,, McMahan, H. B., Mironov, I., Talwar, K., and Zhang, L. (2016). Deep learning with differential privacy. In Proceedings of the 20/6 ACM SIGSAC conference computer and communications security, pages 308-318.
Antunes, R. S., André da Costa, C., Küderle, A., Yari, I. A., and Eskofier, B. (2022). Federated learning for healtheare: Systematic review and architecture proposal. ACM Transactions on Intelligent Systems and Technology (TIST), 13(4):1-23.
Bae, Ng. N., Lo, A., Ghassemi, M., and Grosse, R. B. (2022). If influence functions are the answer, then what is the question? Advances Neural Information Processing Systems, 15:19-9117796.
Balunović, M., Dimitrov, D. I., Staab, R., and Vechev, M. (2022). Bayesian framework for gradient leakage. ICLR.
Chen, C. and Campbell, N. (2021). Understanding training-data leakage from gradients in neural networks for image classification. arXiv preprint arXiv:21 101178
Dayan, I., Roth, H. R., Zhong, A., Harouni, A., Gentili, A., Abidin, A. Z., Liu, A., Costa, A. B., Wood, B. J., Tsai, C.-S,, et al. (2021). Federated learning for predicting clinical outcomes in patients with covid-19. Nature medicine, 27(10):1735-777
Deng, J., Wang. Y., Li, J., Shang, C., Liu, H.,, Rajasekaran, S., and Ding, C. (2021). Tag: Gradient attack on transformer-based language models. arXiv preprint arXiv:2103.0m6
Devlin, J., Chang. M.-W., Lee, K., and Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 1810.04880
Dwork, C. (2006). Differential privacy. In Automata, Languages and Programming: 33rd Interna- lional Colloquium, ICALP 2006, Venice, Itlly, July 10-14, 2006, Proceedings, Purt II 33, pages 1-12. Springer.
Fan, L., Ng, K. W., Ju, C., Zhang, T., Liu, C., Chan, C. S., and Yang, Q. (2020). Rethinking privacy preserving deep learning: How evaluate and thwart privacy attacks. Federated Learning: Privacy and Incentive, pages 32-50.
Fowl, L. H., Geiping, J., Czaja, W., Goldblum, M., and Goldstein, T. (2022). Robbing the fed: Directly obtaining private data in federated learning with modified models. In International Conference on Learning Representations.
Gao, W. Guo, S., Zhang, T., Qiu, H., Wen, Y., and Liu, Y. (2021). Privacy-preserving collaborative learning with automatic transformation search. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1 114-123.
Geiping, J., Bauermeister, H., Dröge, H., and Moeller, M. (2020). Inverting gradients-hov easy is it to break privacy in federated learning? Advances in Neural Information Processing Systems, 33:16937-16647.
Gemulla, R., Nijkamp, E., Haas, J., and Sismanis, Y. (2011). Large-scale matrix factorization with distributed stochastic gradient descent. In Proceedings of the I 17th ACM SIGKDD international conference on Knowledge discovery and data mining. pages 69-77.
Glorot, X. and Bengio, Y. (2010). Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference artificial intelligence and statistics, pages 249-256. JMLR Workshop and Conference Proceedings.
Grazzi, R., Franceschi, L., Pontil, M., and Salzo, S. (2020). On the iteration complexity of hy- pergradient computation. In International Conference on Machine Learning, pages 3748-3758. PMLR.
Guo, C., Karrer, B., Chaudhuri, K., and van der Maaten, L. (2022). Bounding training data recon- struction private (deep) learning. International Conference on Machine Learning, pages 8056-8071. PMLR.

---

Haim, N., Vardi, G., Yehudai, G., Shamir, O., et al. (2022). Reconstructing training data from trained neural networks. Advances in Neural Information Processing Systems.
Hampel, F. R. (1974). The influence curve and its role in robust estimation. Journal of the american statistical association, 69(346):383339) 333.393.
Hannun, A., Guo, C.,, and van der Maaten, I.. (2021). Measuring data leakage in machine-learning models with fisher information. In Uncertainty in Artificial Intelligence, pages 760-770. PMLR.
Hatamizadeh, A., Yin, H., Molchanov, P., Myronenko, A., Li, W., Dogra, P., Feng, A., Flores, M., Kautz, J., Xu, D., et al. (2021). Towards understanding the risks of gradient inversion in federated learning.
Hatamizadeh, A., Yin, H., Roth, H. Li, W., Kautz, J., Xu, D., and Molchanov, P. (2022). Gradvit: Gradient inversion of vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10021-10030
Hayes, J., Mahloujifar, S., and Balle, B. (2023). Bounding training data reconstruction in dp-sgd. arXiv preprint arXiv:330.007225.
He, K., Zhang, X., Ren, S., and Sun, J. (2)1 Deep residual learning for image recognition. arxiv 2015. arXiv preprint 1512.033 14.
He, K. Zhang, X., Ren, S., and Sun, J. (2015b). Delving deep into rectifiers: Surpassing human- level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pages 1026-1034.
Huang, Y., Gupta, S., Song, Z., Li, K., and Arora, S. (2021). Evaluating gradient inversion attacks and defenses federated learning. Advances in Neural Information Processing Systems, 34:7232-7244
Huang, Y., Song, Z.,, Li, K., and Arora, S. (2020). Instahide: Instance-hiding schemes for private distributed learning, In International conference on machine learning, pages 4507-4518. PMLR.
Jeon, J., Lee, K., Oh, S., Ok, J., et al. (2021). Gradient inversion with generative image prior. In Advances in neural information processing systems.
Jin, X., Chen, P.-Y., Hsu, C.-Y., Yu, C.-M., and Chen, T. (2021). Cafe: Catastrophic data leakage in vertical federated learning. Advances in Neural Information Processing Systems, 34:994-1006.
Kaissis, G., Ziller, A., Passerat- Pareeaat-Palmbac,, J., Ryffel, T., Usynin, D., Trask, A., Lima Jr, I., Mancuso,
J., Jungmann, F., Steinborn, M.-M., et al. (2021). End-to-end privacy preserving deep learning on iulti-inntitutionall medical imaging. Nature Machine Intelligence, (66):473-444
Koh, P. W. and Liang, P. (2017). Understanding black-box predictions via influence functions. In International conference machine learning, pages 1885-1894 PMLR.
Krizhevsky, A., Hinton, G., et al. (2009). Learning multiple layers of features from tiny images.
LeCun, Y. (1998). The mnist database of handwritten digits. http://yamm. lecun. com/exdb/mnist/.
LeCun, Y., Bottou, I.., Bengio, Y., and Haffner, P. (1998). Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):278--2224
Li, H.. Li, C., Wang, J., Yang, A., Ma, Z., Zhang, Z., and Hua, D. (2023). Review security of federated learning and its application in healthcare. Future Generation Computer Systems, 144:271-290.
Lin, C.-Y. (2004). Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74-81.
Long, G., Tan, Y., Jiang, J., and Zhang, C. (2020). Federated learning for open banking, Federated Learning: Privacy and Incentive, pages 240-254. Springer.
McMahan, B., Moore, E., Ramage, D., Hampson, S., and y Arcas, B. A. (2017). Communication- efficient learning of deep networks from decentralized data. Artificial intelligence and statistics, pages 1273-1282. PMLR.
12

---

Merity, S., Xiong, C., Bradbury, J., and Socher, R. (2016). Pointer sentinel mixture models. arXiv preprint ar
Pan, X., Zhang, M., Yan, Y., Zhu, J., and Yang, Z. (2022). Exploring the security boundary of data reconstruction via neuron exclusivity analysis. In 31st USENIX Security Symposium (USENIX Security 22), pages 3989-4006.
Pruthi, G., Liu, F., Kale, S., and Sundararajan, M. (2020). Estimating training data influence by tracing gradient descent. Advances in Neural Information Processing Systems, 18:11920-19990
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. (2019). Language models are unsupervised multitask learners.
Roth, H. R., Chang, K., Singh, P., Neumark, N., Li, W., Gupta, V., Gupta, S., Qu, L., Ihsani, A., Bizzo, B. C., al. (2020). Federated learning for breast density classification: real- world implementation. In Domain Adaptation and Representation Transfer. and Distributed and Collaborative Learning: Second MICCAI Workshop. DART 2020, and First MICCAI Workshop. DCI. 2020, Held in Conjunction with MICCAI 2020, Lima, Peru, October 4-8, 2020, Proceedings 2. pages 181-191. Springer.
Scheliga, D., Mäder, P., and Seeland, M. (2022). Precode-a generic model extension to prevent deep gradient leakage. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1858.
Sun, J., Li, A., Wang, B., Yang, H., Li, H., and Chen, Y. (2020). Provable defense against privacy leakage in federated learning from representation perspective. arXiv preprint arXiv:2012.06045
Wang, F., Hugh, E., and Li, B. (2023a). More than enough is too much: Adaptive defenses against gradient leakage in production federated learning. In INFOCOM.
Wang, Z., Lee, J., and Lei, Q. (2023b). Reconstructing training data from model gradient, provably. In International Conference on Artificial Intelligence and Statistics, pages 6595-6612. PMLR.
Wei, W., Liu, L., Wut, Y., Su, G., and Iyengar, A. (2021). Gradient-leakage resilient federated learning. In 2021 IEEE 41st International Conference on Distributed Computing Systems (ICDCS), pages 797-807. IEEE.
Wen, Y. Geiping. J.. Fowl, L., Goldblum, M., and Goldstein, T. (2022). Fishing for user data in large-batch federated learning via gradient magnification. arXiv preprint arXiv:2202.005800
Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q.. Macherey, K., Klingner, J., Shah, A., Johnson, M., Liu, X., Lukasz Kaiser, Gouws, S., Kato, Y., Kudo, T., Kazawa, H., Stevens, K., Kurian, G., Patil, N., Wang. W., Young, C., Smith, J., Riesa, J., Rudnick, A., Vinyals, O., Corrado, G., Hughes, M., and Dean, (2016). Google's neural machine translation system: Bridging the gap between human and machine translation.
Xu, J.. Glicksberg, B. S., Su, C.,, Walker, P., Bian, J., and Wang. F. (2021). Federated learning for healthcare informatics. Journal of Healthcare Informatics Research, 5:1-19.
Zhang, H., Cisse, M., Dauphin, Y. N., and Lopez-Paz, D. (2017). mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09411
Zhang, H., Hong. J., Dong, F., Drew, S., Xue, L., and Zhou, J. (2023). A privacy-preserving hybrid federated learning framework for financial crime detection. arXiv preprint arXiv:2302.03654.
Zhang, R., Isola, P., Efros, A. A., Shechtman, E.,, and Wang. O. (2018). The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and patlern recognition, pages 586-595.
Zhao, B., Mopuri, K. R., and Bilen, H. (2020). idlg: Improved deep leakage from gradients. arXiv preprint arXiv:2001.02617 022600
Zhu, J. and Blaschko, M. (2021). g-gap: Recursive gradient attack on privacy. In International Conference on Learning Representations.
7hu, L., Liu, 7., and Han, S. (2019). Deep leakage from gradients. Advances in neural information processing systems, 32.
13

---

A Method
A.1 Other Efficient Evaluation Techniques
The techniques for efficiently evaluating implicit gradients can be referred to (Koh and Liang, 2017; Grazzi al., 2020). As computing the inverse second-order derivatives is the most computation- intensive operation, we will focus on it. Here, we briefly summarize two supplementary techniques introduced in Section 3.1.
Conjucate gradient. In Section 3.1, we use the trick of least square to compute the (JJT)-1J8
When JJJ > O, we can solve the least square problem by the conjugate gradient (CG) method, which only needs (dd) time to converge. However, the algorithm will be unstable when the matrix JJ is ill-conditioned. As observed in Fig. 5, the JJ is likely to be ill-conditioned for deep networks. But here we provide this alternative for linear models such that J can be evaluated faster.
Neumann series. We can leverage the Neumann series to compute the matrix inverse. By the Neumann series, we have (JJJ)-1J6
lim,->oe Ei=o(! y Let S, - Ei
y and - Jő. Then the computation can be done by iteration St41 (I JJ )8t J6 which only includes Jacobian-vece vector products.
B Proofs
B.1 Proof of the Approximation by Implicit Gradients
Here, we provide the proof for aG.(go)
(JJJ)-1J. Recall Eq. (1) as
8gu
x²
= G.
arg min (r - VoL(x,0) - gll
(8)
The stationary condition of the minimization gives
ÖL, (x":g) 0.
Or
Given a small perturbation Ag + 0 the gradient, can estimate corresponding perturbation
Ag. 0 as a function of Ag. Thus, can approximate Un" by asg Use Taylor expansion to show approximately
aA.-,
8g
(Tg(x";g) Or + 8 (1(((=)))) (L1(x":9) 2x
0g0r Ag +
A.- 00
8 (1("99)))
0L,(2" Ox
A -
Dg
Ox
dr,
(0)))(10)))
(8 L
o2 PL)(x":9) Ogdx
0g
Ox
where we drop higher-order perturbations. The above derivations can be rigorously proved using the Implicit Function Theorem. Since Oz
(L sg) 2(VoL((" VoL(x", we can derive
82 (x Og0x
- VoL
and
0 L1 22 -0 -o VoVV
dz
As To = x- = Gr(gu) and gu VoL(x" 0), we can substitute them to obtain
82 (r 2V, VoL(Zo. = -- (go)) 0),
(10)
dgdro
8 (x 2(g0 V) 2.J
(11)
On,²
14

---

Let 9o- Combine Eqs. (9) to (11) to get
(Gr( = (J
Ogo
B.2
Proof of Theorem 3.1
Before we prove our main theorem, we prove several essential lemmas below.
Lemma
B.1.
V VoL(2,
0)| v. v. (o
S (oz (Voo((201
PJ (
x--10|
Proof.
v. VoL((, 0)1 V. VV. 0)1²
V o [V V. VoVo
+V Vo, ( V. V ( 0)
- VV VL 8) V (x, - VoL(2', 0)1 + V VoL(x',0) 0) v. V.V,, 8) VV 0)
Using Assumption 3.4 and 3.5 directly lead to
V. Φ Vo V VoL((' 0)]| s (n7 VV
(2), PJ VoL(E' 0)1l) 1| - 2'
Lemma B.2. V - 0))9 s rM || x'1
Proof. By Assumption 3.4, have
( ()) - (V M(2' 0)9) VV V VoL((' 8) |g
H4 [g | - -
Lemma B.3. The inversion loss L)(r:)) defined satisfies [V,L)(< v=b((r'))) || x'll where
pla, = 14 + / p. |gg||
(12)
Proof. Since
VeL((z:)) 2V (x 0)
2V - 0)g.
we can derive
V (F (=L)('
= V V (I. 0][ V. VoL(2' 0]|
V.V. - V.V. 0) 9f
< 2 V VoL(a, 2 V. VoL(E' |/11 +2 [V~ |
L(x,
- V. Vo2
By Lemma B.I and Lemma B.2, we have
V [ 0)1 v. V (=' ,0) S /vl || - -
V L(x, 0)g) (. L(x' 0)g) < n2 | - x'
where f4 = PL V (z 0) (2 0) and μ2 = mu |lg|] Let μ = 2411 + -22. Then can get
V (= v.Li(x':)) s --1x=~211
15

---

Theorem B.1 (Restated from Theorem 3.1). Let Do be the private data and 9u - VoL(xo, 8) be its corresponding gradient which treated as a constant. If Assumption 3.4 and 3.5 hold, then the square root of the recovery RMSE satisfies:
J|
I Gr + +6)11 2
(13)
Mr |J] + 2p 1990
51
where J V. VoVoL
Proof. Utilize the stationary condition V (=( g) = 0 and Lemma B.3 to obtain
[[ S μ(x, g. 0, 9) || =اll= Vr.
As zo is the private sample whose gradient go _ (oo, 0). then we have
11000
Ggo + +0]l 2
V 90 +]]
plzo, x 96 0, go + +8)
Because
V L1 (zo: + 5) = 2V V L () 8)
2V & (x)
we can attain
2
|100 - Gr + 6)11 2 1
[V
With VoL(I
90 0) = 90 6, we can obtain
1(00, -:: 0,000 + 6) 2ur V. VoL 0)1 2p, L( 0)| 2n, 1190 + 8||
24r ||||| + 4 l190 + +61
C Experiments
C.1 Experimental Details
Model architectures. The linear model use is matrix that maps the input data into vector. The LeNet model is a convolutional neural network with 4 convolutional layers and fully connected layer. We use the modified version following previous privacy papers Sun al. (2020), whose detailed structure is in Table 1. ResNet18 is a popular deep convolutional network in computer vision with match-normalization and residual layers (He al., 2015a). Cross entropy loss is used the loss function in all the experiments.
Experimental settings. We conduct two kinds of attacks in our paper: DGL. and GS attacks. The learning rate of the two attacks is 0.1 and we use Adam as the optimizer. To consider more powerful attack, only single image is reconstructed in each inversion. When inverting LeNet, we uniformly initialize the model parameters in the range of [-0.5, 0.5) as (Sun et al., 2020) to get a stronger attack. When inverting ResNet18, we use the default initialization method in PyTorch and follow Huang et al. (2021) to use BN statistics as an additional regularization term to conduct a stronger attack.
Table 1: A modified version of LeNet. Conv represents convolutional layer. FC means a fully- connected layer.
Layers
Conv(in_chanle channels=3, hot__hhnnels=122 kernel_size=5)
Conv(in_chaneels=11 channels=12, nut_channels=12. kernel_sizees size=_)
Conv(in_channele 12, out_channels=12, kernel_size=5)
Conv(in chwninllsc1222222 out channels= kernel size=5)
Flattern
FC(out features= 10)
16

---

10
6x10
4x10:
Google BLEU
feature MSE
ROUGE-1
ROUGE-L
3x10-1
10-3
10-
10-1
10-1
10
10
10
10
10
10:4
10
10
10
10th
10
10
10
10t4
10
10
Lib
To
Tp
(a) ROUGE-1
(b) ROUGE-L
(c) Google BLEU
(d) Feature MSE
109
6x10-
Google BLEU
feature MSE
4x10-
ROUGE-1
ROUGE-L
10
Ipp
7
Apb
Zib
(e) ROUGE-1
(f) ROUGE-L
Google BLEU
(h) Feature MSE
Figure 9: Evaluation of Tt on BERT (top) and GPT-2 (bottom). A darker color means a larger noise variance. Four metrics are used to evaluate the semantic similarity between the original text and recovered text. Iz is linearly correlated to the four semantic metrics, which means Iis can be used to estimate the privacy risk of the private text.
C.2 Empirical Validation Language Data
We evaluate the proposed I metric on BERT (Devlin et al., 2018) and GPT-2 (Radford et al., 2019), which are popular language models in natural language processing. We use TAG (Deng et al., 2021), which is an attack on Transformer-basee language models based on the L1 and L2 distance between the original gradient and the dummy gradient, and follow the code in https: //github. cttibb/.oom//aaa/ SeiraaGeipinng/bee update the input embedding 12,000 times. We randomly sample 70 sentences from WikiText- 103 (Merity et al., 2016) the private text. We use ROUGE-1, ROUGE-L (Lin, 2004), Google BLEU (Wu et al., 2016) and feature MSE to measure the semantic similarity between the original text and the recovered text. ROUGE-1 measures the overlap of 1 I-grams in the original and recovered text, while ROUGE-L measures the length the longest common subsequence between two sentences. While ROUGE metrics calculate the reconstruction recall, the Google BLEU score uses the output the smaller value of the precision and recall of the original and recovered text and has broader range of the overlap of n-grams, where consistent with our assumptions, we include a continuous metric, the feature MSE, which measures the distance between the final layer's feature of the original and recovered text.
We use the default setting in the code and iteratively
1, 3, 4. Since the above three metrics discrete and not
The results are presented Fig. 9. A darker color indicates a larger noise variance. For each noise variance, randomly sample the perturbation from mean Gaussian distribution times and repeat this for 3 different random seeds. shows that I'F is correlated to these four metrics, which means I'F can be used estimate the privacy risk of text datasets with large language models. The correlation of BERT and GPT-2 between the four metrics has a similar mode. Though ROUGE-1, ROUGE-L and Google BLEU measure the structural similarity of sentences, which consists of discrete tokens, I presents a clear correlation. For the feature MSE, although I&F has less distinguished correlation with feature MSE, it can still be utilized estimate the privacy risk.
C.3 Efficiency
To show the efficiency of computing the I'F values, we compare it with the GS attack on randomly- picked CIFARIO images with ResNet18. maximal eigenvalue via the power iteration method, we compare the time required for the power iteration against the inversion loss of GS to converge. We notice that the convergence depends on the initialization of the power iteration and the learning rate for DGL. Thus, repeat power iteration with
the major complexity comes from evaluating the
17

---

12.8
0.65
0.60
digenvalus
12.6
12.4
Inver Sersson
0.50
12.2
0.40
12.0
0.35-
0.30
1.0 1.5 2.0 2.5 3.0 3.5 4.0 4.5
20 40 60 no 160 120 140 1ho 0
running time (sec)
running time (sec)
(a) Power iteration
(b) Inversion attack
Figure 10: Evaluation of the efficiency of computing (J) (our method) by power iteration and inversion attack by minimizing inversion loss (Lr). Colors in (a) indicate different seeds. Darker colors in (b) indicate larger learning rates. Our method using power iteration can converge faster than direct inversion attacks.
10a
Ze
10a
Io
Inversion
Inversion
time (sec)
10a
100
time (snc)
10a
10
10a
10a
10"
(a) DGL attack
(b) GS attack
Figure 11: Comparison of the efficiency of computing Its (our method) by power iteration and inversion attack by minimizing inversion loss (L1). Blue bars indicate the time of computing Zı while orange bars indicate the time of minimizing inversion loss by DGL and GS. The time ratio of computing Its model-dataset pairs sorted by the model scales: MLP-MNIST,
minimizing inversion loss is present above the orange bars. The x-axis
MNNNT-,MIST, LeNet- CIFAR 10, RN18-MNIST, EN18-CIFARID 10, EN34-CIFAR 10, RN5O-CIFAR 10, RN101-CIFAL 10, RN152-CIFAR 10, RN152- Imageee. For large models and datasets, where minimizing inversion loss needs huge computation overhead, In can provide an efficient estimation of the privacy risk.
5 different seeds. For the inversion attack, we evaluate multiple learning rates (1, 0.5, 0.1, 0.05, 0.01) show the fastest convergence. Each experiment repeated 5 times with different random seeds. As shown in Fig. 10, the power iteration method can converge the maximal eigenvalue within 50 iterations (5 seconds most). In comparison, 150 seconds to fully converge, which is 20 times larger than the power iteration method. Thus, our method can give an accurate and fast approximation of the recovery MSE without the exhaustive whole inversion process.
inversion loss demands 3000 more iterations in
In Fig. 11, we compare the computation cost of computing Zn with minimizing inversion loss. We show that for almost all the models and datasets we evaluate, the time ratio is larger than 1, which means it is more efficient to compute I than minimize the inversion loss. It indicates that our method is a more efficient way to estimate the privacy risk for most models and datasets, than the empirical inversion attack. Another key point is that, large models and datasets, such as models larger than RN18 with CIFAR 10 or ImageNet, the time ratio is even much larger than 500. When the time consumption of inversion attacks on these models and datasets is huge (about 16 minutes even longer), method significantly reduces the computation overheads for estimating the privacy risks.
18

---

(a) A = 3.90
(b)
(c)
(d)
À = 0.12
A = 0.03
X = 0.0006
Figure 12: Same perturbation sizes but different protection effects by different eigenvectors of LeNet. Recovered CIFARIO images associated with different eigenvectors are present. When perturbing with eigenvectors with smaller eigenvalues, the recovered images are more noisy.
(a) 2.60
(b) 2.71
(c) 2.39
(d) 3.97
(c) 0.15
(f) 0.28
(g) 0.32
(h) 0.33
Figure 13: Same perturbation sizes but different protection effects by different eigenvectors of ResNet18. Recovered CIFARIO images associated with different eigenvalues are present. When perturbing with eigenvectors with smaller eigenvalues, the recovered images are more noisy and lack some semantic information.
C.4 More Visual Results
More images of unequal perturbations. We present in Fig. 12 the recovered CIFAR10 images when perturbing the gradient with eigenvectors with different eigenvalues. When perturbing with eigenvectors with smaller eigenvalues, the recovered images are more noisy, which is consistent with our former observation.
We also present the unequal perturbations with different eigenvectors ResNet18 on the CIFARIO dataset Fig. 13. Even with the same perturbation scale, the eigenvectors with larger eigenvalues provide stronger protection, where the corresponding recovered images are more noisy and lose some semantic information.
More images of unfair privacy protection. We show more results of unfair privacy protection in Fig. 14. The images of digits 5 and 8 can still be recognized by their outlines, while images of digits 7 and 9 are unrecognizable noise.
We also show the unfair privacy protection of ResNer18 on the CIFAR10 dataset in Fig. 15. In this experiment, we also observe large variance of recovery MSE among samples, indicating sample-wisc unfairness. At the class level, we still can find gradients of few classes to be easily inverted. For example, class 8 has most MSES lower than the average value.
D I'F with Gradient Pruning Defense
We present the relationship between the RMSE and Ins in Fig. 16. The y-axis RMSE and the x-axis Lus It shows Ia be used to estimate the worst-case privacy risk with gradient pruning defense.
E Comparison of I'F with More Metrics
MSE is pixel-wise distance that lacks semantic and structural information. evaluate the effec- tiveness of I2F on more metries, we consider SSIM and LPIPS (Zhang et al., 2018) to measure the
19

---

7
8
9 (d)
(
(b)
(c)
(f)
(g)
(h)
Figure 14: Original (top) and corresponding recovered (bottom) images of LeNet on the MNIST
dataset. The gradients perturbed with Gaussian noise of variance 10-3 The defense is unfair as images of digit and digit 8 can be recognized by the outline.
6.14
0.10
MSE
04
5006
0.3
0.2
MSB
Figure 15: The sample-wise and class-wise statistics of the GS MSE on the CIFARIO dataset of ResNet18. The purple lines indicate the average values. Large variances observed among samples. The original (first row) and recovered (second row) images for the well- and depicted the bottom. The defense is unfair (ship), are more vulnerable to inversion attacks.
classes classes, e.g., class 7 (horse) and class
8858
2
10a
10
0.2
10-2
0.2
10"
(a) RNIS-CIFARIO-GS
(b) RNI8-CIFARIO-DGL
(c) RNI8-MNIST-GS
(d) RNI8-MNIST-DO
109
10
10a
10
0.4
10 10a
1 10 10
16
10
10
10a
10
10
(c) LeNet-MNIST-GS
(f) LNNSS-MIIST-
(g) LINAR-CTFARIO (h) LeNGE-CIFARIO-DGI
Figure 16: RMSE (y-axis) vs. Ii (x-axis) with gradient pruning. A darker color indicates smaller pruning ratio. It shows that Typ is good estimator of RMSE.
20

---

structural similarity and semantic distance between the original and recovered images, respectively. The relationship between SSIM and LPIPS shown in Fig. 17. Since Lι, aims to lower bound the privacy risk in terms of RMSE, Lts does not have the same scale SSIM and LPIPS. Nevertheless, In also has positive correlation between SSIM and LPIPS, which implies that it a good estimator for the structural similarity and semantic distance between the original and recovered images.
10"
10"4
10
10-2
10
10
10
10a2
107
(a) RNI8 CIFARIO-GS
(b)
RNIS-CIFARIO
(c) LeNe--MNIST-GS
(d) LeNS--MNIST-DLL
DGL
0.8
0.1
0.1
0.2
0.2
0.1
0.02
0.1
16t2 10
10P
10a4 10a
10"
16 10
10e
10"2
10th
10
(e) RNI8 CIII8-IFAAII-GS
(f) DGL
(g) LeNNe-MNIST-GS
(h) LeNee-MNIIT--OR
Figure 17: I, is positively correlated with these two metrics and is good estimator for the structural similarity and semantic distance between the original and recovered images. Darker color indicates higher variance. Top ((a)-(d)): SSIM (y-axis) vs. Iιs (x-axis). Bottom ((e)-(h)); LPIPS (y-axis) vs. Tab (x-axis). A higher SSIM and lower LPIPS indicate higher privacy risk.
F Dynamies of Tιb During Training
Previous existing empirical results show that privacy risk decreases by training epochs (Balunović et al., 2022; Geiping et al., 2020). We evaluate the dynamics of Iy. RMSE with DGL attack and RMSE during training in Fig. 18. It shows that as the training epoch increases, the Ia also has an increasing trend. While the RMSE of RNI8 on the CIFARIO dataset has a similar trend as Lıo, that of LeNet on the MNIST dataset has significant rise the epoch 60, which is due the slower learning speed of LeNet than RN18. Moreover, almost for all the epochs, there is a sample with low Iy, which again emphasizes the unfairness in privacy protection.
G The Impact of on Efficient Matrix Inversion
In Fig. 19, we study the impact of e on efficient matrix inversion proposed in Section 4. We evaluate the impact the LeNet with the MNIST dataset. The y-axis is the RMSE. I (matrix inversion) is calculated as defined Eq. (4). Zy (matrix norm) is calculated as defined in Eq. (5). It shows with C [1, 10), In is lower bound of the RMSE. It also shows that Tιp is accurate estimator of F Thus, we can directly use I in practice to lower bound the privacy risk to avoid fine-tuning the c.
H Discussion
H.1 Validity of Assumption 3.4-3.5
We make assumptions about the Lipschitz continuous Jacobian and gradient Assumption 3.4 and Assumption 3.5, respectively. These two assumptions are not necessary for I2F but are only used to provide theoretical validation of I' when the noise á not infinitesimal. To discuss the validity of these two assumptions in practice, calculate the value of μu and μz of LeNet with two datasets.
21

---

10
0.6
10
2
0.4
10
1
0.3
10
0.2
10-3
0.2
0.1
0
20
40
60
80
0
20
80
0
20
40
(a) RNI8 CIFARI0
106
100
100
10
10
10
10
10
10-2
10-2
10-2
0
20
40
60
80
0
20
40
60
80
0
20
40
60
80
(b) LeNet on MNIST
Figure 18: Privacy risks decrease by training epochs (æ-axis). Different colors indicate different samples. The y-axis from left right: Zg, RMSE DGL attack, RMSE w/ GS attack whose smaller values indicate lower risks. The blue line indicates the mean value and the shadow is the variance (some outliers are dropped). The noise is sampled from a Gaussian distribution with mean zero and variance 10-3
For the CIFAR1O dataset, PL 0.5014 and μ.l 1.7 10-13, For the MNIST dataset, Pz 0.7192 and 3.7 x 10 13 These values are not so large that they are reasonable in practice.
H.2 Extension of I'F to the GS Attack
The derivation of 12F is considered in the DGL attack as defined in Eq. (1), but I can also be applied the GS attack. Note that the minimizer of the DGL attack is one solution the GS attack. That means the GS attack can be attained by an optimal DGL. attack which is our assumption. Therefore, the DGL attack- based theorem is applicable to the GS attack.
Empirically, we evaluate Zn on GS attack in Figs. and 16. shows that Izs is linearly correlated to the metrics of RMSE, which proves the utility of I'F under GS attack.
H.3 Extension of I'F with Prior Knowledge
Our theorem of I2F can be extended to take into account the prior knowledge. Consider the inversion optimization problem with prior knowledge as min, (x; g) L, (2:; g) + (x where (a constrains in the prior space C' and L₁ (2; g) [VoL(z; 0) all defined in Eq. (1). Then the optimization problem can rewritten as minzeg Lı(x; g). Thus, as long as the original image do is in the feasible region defined by Ic(x), our Assumption 3. and theorems are also applicable. Intuitively, good regularization should satisfy the requirement, otherwise, will unreasonably reject the correct recovery.
22

---

8
10
10"1
10
10-1
10
10"2
10
10
10 10 10
10
10 10
10 10 10 10 (matrix inversion)
10 10 10
(matrix inversion)
Z (matrix inversion)
In (matrix norm)
(a) DGL (4 = 0.1)
(b) DGL 1)
(c) DGL (e = 10)
(d) DGL (In))
10-1
10
10-2
10
10-2
10
10 10 10 10e
10-5
10
10 (matrix inversion)
10
10
10 10th 10
10-3
10 10 10-7
(matrix inversion)
(matrix inversion)
In (matrix norm)
(e) GS (c =c= 0.1)
(1) GS (c = 1)
(g) GS (c = 10)
(h) GS (Ia.)
Figure 19: The impact of the value of is evaluated on LeNet with the MNIST dataset. The y-axis is the RMSE. (d)) DGL attack. (h)) GS attack. is observed that I (matrix inversion) effective with E [1. 10] but not = 0.1. shows that (1) there exists a range of c where Iis can lower bound the RMSE; (2) Tu is accurate estimator of I2F, thus can avoid fine-tuning
H.4 Discussion with Prior Works
Closest to our work, Hannun et al. (2021) provided second-order worst-case metric for analyzing privacy attacks. However, our work provides novel contributions both on technique and implications which essentially root from the proposed I2F metric.
Technical difference. First, we focus different scope against (Hannun al., 2021). Hannun et al. (2021) proposed Fisher information loss (FIL) to measure the information leakage risk in the context of model inversion and attribute inference, e-g., only the attribute inference is considered in their experiments. Instead, we evaluate the privacy risk under gradient inversion attacks. Second, our metric is more scalable and applicable large models. For example, in Eq. (18) (Hannun et al., 2021), the inverse of the Hessian matrix needs to be calculated even when quantifying the information leakage of only one sample, which inefficient and intractable for large models. Because of computation inefficiency, only linear regression and logistic regression models considered in their theories and experiments. Instead, we verify the feasibility of I2F in much larger models like 55 on ImageNet in Fig. 3.
Our new findings. First, though the unfairness of information leakage of different samples was discussed in (Hannun et al., 2021), we investigate the issue in a different attack method and justify the commonness of the unfairness different attacks. Second, we additionally provide other insights than (Hannun et al., 2021). For instance, the influence gradient inversion of different initialization methods is studied. We also find the influence of perturbations is not equivalent even in the same noise scale. We believe these insights are also critical to the privacy and security community, especially in the area of gradient inversion.
Besides, (Guo et al., 2022; Hayes et al., 2023) propose to bound the reconstruction attack in terms of the attack success rate and the expectation the L2 distance between the recovered and original image. Nevertheless, their conclusions are based on the differential privacy (DP) quantification framework so it is hard analyze the influence of other defense mechanisms such as gradient pruning and arbitrary noise. Also, they bound the privacy risk from the statistical sense raised by the randomness of DP, while our work can evaluate sample-wise worst-case privacy risk at any time during the model training. Moreover, they assume the access of the attacker to the remaining samples (except for the privacy sample) or the multi-round gradient, which not practical in real gradient inversion scenarios.
23

---

H.S Discussions about the Worst-case Assumption
Our work mainly built upon the Assumption 3.1 that there is only unique minimizer for Ly (z; g) given a gradient vector 9. Because of the hardness of optimizing a non-linear objective in attack (Eq. (1)), the worst-case may not be reachable in practice. Here. we discuss when the assumption has to be relaxed and why our method is still applicable. In addition, we emphasize that a stronger attacker exists theoretically, resulting in the necessity of a worst-case assumption.
Non-bijective inversion mapping G)) G,((g) could be non-bijective when the loss function is non-convex. In other words, given the same g. the output of (g) could include multiple choices. We want to argue that this case does not conflict with our assumption. Consider an attack given the exact gradient of sample. Note that the sample itself is solution to Eq. (1), Thus, given the exact gradient of sample. the attack can exactly recover the sample. Even if the solution is non-umique, we can still essentially assume the attack can attain the sample in the worst case.
Optimizing the gradient inversion objective may not converge to the original image. Note that the original image is always optimal solution for the inversion loss in Eq. (1). Even though the convergence is not guaranteed, there always exists an algorithm that can converge to the original image. To our best knowledge, there is evidence to show the attack cannot approach the worst case where the original input is recovered. Instead, empirical results have shown that the images can be recovered almost perfectly (Geiping et al., 2020). Thus, due to the sensitivity of privacy, worst-case assumption is necessary to strictly bound possible privacy risks with arbitrary strong attacks, which is commonly imposed by the literature (Dwork, 2006; Abadi al., 2016).
I Realistic Impact
Federated learning (FL) (McMahan et al., 2017) is popular distributed training paradigm that benefits from the data and computation sources from multiple clients. As a principle of FL, clients will upload the local gradient based the private data to the server for the concerns of data privacy and safety, instead of directly sharing the private data of each client. However, recent works (Geiping et al., 2020; Zhu et al., 2019) show that attacker, who has access to the local gradient (e.g., malicious server), can leverage the local gradient to recover the private data of the clients, which we call Deep Gradient Leakage (DGL). Even with large models like Transformer, the attacker can still successfully recover the private data given the gradient (Hatamizadeh et al., 2022).
Auditing potential privacy risks is essentially desired for seivacy- applications, including but not limited to finance (Long et al., 2020), healthcare (Antunes et al., 2022; Xu et al., 2021), and clinical data (Dayan et al., 2021; Roth et al., 2020). Such privacy concerns have been discussed extensively in previous work. For instance, Zhang et al. (2023) raises the privacy concern of FL. in financial crime detection while (Kaissis et al., 2021; Li et al., 2023) discusses it in the medical and healthcare applications, respectively.
To echo the demands for privacy risk auditing, we provide a fundamental tool for bounding the worst-case risks of DGL. We show multiple insights, for example, the unfairness of privacy protection using random noise defense. Thus, we expect our work can call the attention of the community to the privacy concern raised by DGL, especially the worst-case instance-level privacy risk. Moreover, We expect I2F to be keystone for designing more powerful defense mechanisms.
24


Reviews:
[{"review_id": "3tH8mZNwIY", "reviewer": "Reviewer_NgNT", "summary": "This paper aims to understand how private information leaks from gradients in model training.\nTo this end, the authors propose to use influence analysis to analyze how gradient perturbations can affect the quality of samples reconstructed from private gradients.\nSpecifically, they first prove that under some assumptions, the privacy leaked from a perturbed gradient can be strictly characterized by a novel Inversion Influence Function (I2F), which is a function that takes the gradient perturbation as input.\nWith I2F, the authors then gain insights about DGL, such as (1) common random noise (e.g., Gaussian noise) can not provide uniformly fair privacy protection for each gradient, (2) future protection for DGL can be developed based on the eigenvalues of the Jacobian matrices of the model.", "questions": "None.", "limitations": "This paper contributes non-trivially to understanding how privacy leaks from gradients.\nAlthough the empirical analysis could be a little weak, I think this work will significantly contribute to the ML security community and therefore should be accepted.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "This paper makes the first step to address an important and fundamental problem, i.e., decoupling the black-box behavior of data gradient leakage. The authors make non-trivial contributions in various aspects:\n\n1. The authors design a novel influence function named I2F based on non-trivial influence analysis. It can strictly characterize the privacy protection provided by gradient perturbation, which gives the first theoretical result about how to analyze gradient privacy leakage.\n\n2. Beyond theoretical analysis, the authors also conduct empirical studies to justify the feasibility of using I2F to study DGL in practice.\n\n3. Based on I2F, the authors found that the singular values of model Jacobian with respect to single inputs could be a good indicator for representing privacy protection strengths, which then illustrates that the gradient privacy protection provided by random noise (which is commonly used in practice) could be unfair to different training data.\n\n4. The paper first establishes the connection between model Jacobian and gradient privacy protection, which suggests that future defense mechanisms can be designed starting from the singular values of model Jacobian.", "weaknesses": "1. The main weakness is that the experiments could be not comprehensive enough to justify the effectiveness of I2F. Specifically:\n    - Only two datasets, MNIST and CIFAR-10, are involved in the experiments. Furthermore, one of the datasets MNIST could be too simple for illustrating representative results.\n\n    - Only a few models are used in the experiments and most of them could be too small.\n\n2. Suggestion: I think it is not necessary to use $\\mathcal I_{\\mathrm{lb}}$ as a replacement to approximate I2F since the calculation of I2F seems not really costly. Specifically, the main calculation cost would be calculating the matrix inversion of $JJ^T$. Because $JJ^T$ is (assumed to be) positive definite, you can easily use `numpy.linalg.eigh()` to calculate the matrix inverse, which is usually calculation efficient in practice.\n\n======== After Rebuttal ========\n\nScore has been raised based on the authors response.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We really appreciate the affirmation from the reviewer of our contributions and innovations. We are glad to address the concerns as follows:\n\n**W1:** *Only MNIST and CIFAR10 are in experiments. MNIST is a little simple. Also, the models are too small*\n\n**A1**: Thanks for the suggestion. In Fig.5 of the attached PDF, we evaluate our metric on the large model (ResNet152) and large dataset (ImageNet). \n1. For larger models, the MSE is no longer a good metric for the recovery evaluation. Even if state-of-the-art attacks are used and the recovered image is visually similar to the original image in Fig 2(b), the two images are measured to be different by MSE, due to the visual shift: The dog head is shifted toward the left side. To capture such shifted similarity, we use LPIPS [1] instead, which measures the semantic distance between two images instead of the pixel-to-pixel distance like MSE.\n2. Fig.2(a) shows that I2F is correlated to LPIPS using large models and image scales. This implies that **I2F is a good estimator of recovery similarity even for large models and images**.\n\n**W2:** *No need to approximate I2F with I_{lb}, since we can use numpy.linalg.eigh() to calculate matrix inverse.*\n\n**A2:** A: Thanks for the suggestion. It is a good point that we can use numpy.linalg.eigh() to calculate the matrix inverse. However, for high dimensions of parameters and input images, the main challenge involves the computation of the full Jacobian matrix $J$, besides the inverse. Thus, we propose two alternative approaches in “Efficient Evaluation” to efficiently compute $(JJ^T)^{-1} Jv$ where v is a vector, without explicitly obtaining $JJ^T$.\n\n[A] Zhang, Richard, et al. \"The unreasonable effectiveness of deep features as a perceptual metric.\" CVPR. 2018.\n"}, {"review_id": "1NC4yyl8w2", "reviewer": "Reviewer_BBkr", "summary": "This paper proposes to leverage influence function as a tool for understanding and analyzing the privacy risk in gradient leakage by connecting the private gradients with the recovered images. Inversion Influence Function (I^2F) is introduced as an efficient approximation of deep leakage attacks. Theoretical justification is provided for this approximation and empirical analysis is conducted on two image datasets (MNIST and CIFAR10).", "questions": "1. It can be seen from Fig. 2 that Cifar10 samples induce a larger variance compared to MNIST. Is this an artifact of Cifar10’s data distribution being more complex?\n2. What are the variances of the Gaussian noise used to produce Fig. 1 & 2?\n3. How much data is needed to tune the epsilon constant? How much impact does the constant have on the results? E.g., how would I^2F perform if an epsilon constant tuned on a different dataset is used?\n", "limitations": "The limitations and social impact have been discussed in the last section.\n", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. Understanding privacy leakage through gradients is an important and timely topic.\n2. It is an original work and perhaps the first to leverage influence function to perform analysis on deep gradient leakage.\n3. The analysis provides an explanation for sample- and class-wise variance in reconstruction quality (referred to as “unfair privacy” in the paper).\n", "weaknesses": "1. For complex neural networks with non-convex loss functions, the inversion mapping Gr may not be bijective, e.g., there might exist multiple x that induce a similar gradient.\n2. The proposed inversion influence function based on first-order Taylor requires the added perturbation to be infinitesimal to be accurate. Although this is partially justified through Theorem 3.1, it can be seen from Fig. 1 that the lower bound is violated at larger noise values.\n3. For singular Jacobians, a constant needs to be added for numerical stability, which has to be tuned for each specific dataset.\n4. From what the reviewer understands, the performed analysis through influence function only considered an adversary who observes the (noisy) gradients and tries to perform inversion by solving the gradient matching optimization problem. In practice, a sophisticated attacker may leverage prior knowledge to improve the reconstruction. The experiments only involved attacks with no/weak prior (e.g., TV), it would be interesting to see how well the proposed metric approximates the reconstruction quality of a strong biased attacker, e.g., using GAN as prior.\n5. In the batch recovery case, the upper bound by decomposing into individual gradient inversion might be too loose.\n6. Not a weakness but some discussion and comparisons between the proposed lower bound and prior work on lower bounding reconstruction MSE would be nice to have, e.g., [R1] using estimation theory and [R2] using DP-SGD.\n[R1] Guo, Chuan, et al. \"Bounding training data reconstruction in private (deep) learning.\" International Conference on Machine Learning. PMLR, 2022.\n[R2] Hayes, Jamie, Saeed Mahloujifar, and Borja Balle. \"Bounding Training Data Reconstruction in DP-SGD.\" arXiv preprint arXiv:2302.07225 (2023).\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the positive comments on our contribution and novelty toward understanding the gradient inversion attack!\n\n**W1:** For complex NN and non-convex loss func, the inversion mapping $G_r$ may be not bijective\n\n**A2:** Thanks for your comments. Indeed $G_r$ may be not bijective but it is not a weakness of our work, since **it does not conflict with our assumption**. We assume a perfect attack given the exact gradient of a sample.\n- Note that the sample itself is a solution for Eq(1). If the solution of the inversion attack is unique, then it is equivalent to say the attack optimization method for solving Eq (1) is optimal. Note that the sample of the gradient is a solution for Eq(1). Thus, given the exact gradient of a sample, the attack can exactly recover the sample. Even if the solution is non-unique, we can still essentially assume the attack can attain the sample in the worst case. \n\n**W2:** I2F needs Talyor expansion, which requires perturbation to be infinitesimal.\n\n**A2**: Thanks for the question. \n1. Firstly, with a small noise, it is more likely for the attack to recover an image closer to the original one [C]. Thus the privacy risk could be higher, so it is more valuable to bound it. \n2. Second, since large noise can destroy the performance [C, D], a small noise is often a common and better choice so we focus more on the privacy risk under a smaller perturbation. \n3. Third, we also provide a theoretical validation in Sec 3.2 to show the utility of $I^2F$ under non-infinitesimal noise, where we show I2F is still effective as a lower bound even with large noise. \n\n**W3&Q3:** For singular Jacobians, a constant $\\epsilon$ should be tuned. How much impact does the constant have on the results?\n\n**A3:** Thanks for the question.\n1. First, **we don’t need to fine-tune the constant $\\epsilon$ in our experiments (and also in practice) except in Fig.1**. In Fig.1, it shows the extension for singular Jacobians (known as $I$ (matrix inversion)) can be estimated by $I_{lb}$. Thus we can use $I_{lb}$ to evaluate privacy risk without fine-tuning the $\\epsilon$.\n2. The impact of the constant $\\epsilon$ is evaluated in Fig.2 of the attached PDF. There are two main observations. \n    1. First, there exists a range of $\\epsilon$ where I$^2$F can be used to estimate the privacy risk accurately. So the target of fine-tuning $\\epsilon$ is to find an optimal range instead of an optimal value, which makes fine-tuning much easier. \n    2. Second, in the optimal range of $\\epsilon$, $I_{lb}$ is an accurate estimator of I$^2$F, thus the fine-tuning of $\\epsilon$ can be avoided.\n\n**W4:** The experiments only involve attacks with no/weak prior (TV loss), more stronger prior should be used.\n\n**A4:** Thanks for the suggestion. We use the TV loss as a prior when attacking LeNet because the attack is strong enough. While when attacking RN18, we use BN statistics as a stronger prior as explained in lines 527-533. Fig.2 shows the evaluation of the effectiveness of I2F with GS attack with BN prior. It indicates that under a stronger attack with BN prior, I2F can still estimate the worst-case privacy risk.\n\n**W5:** Loose bound of batch recovery\n\n**A5:** Thanks for the question. We follow prior work like [E, F] and investigated the privacy risk of a single sample, and therefore the main focus is on the sample-wise attack. The upper bound in “Batch data” in lines 236-240 is an application of sample-wise attack analysis when the attacker can get the per-sample gradient in a batch instead of the average gradient of the batch. Under such a case, our proposed I2F can be applied to the worst-case batch-wise attack. We agree with the reviewer that a batch-wise extension of I2F is very useful and is an interesting future work.\n\n**W6:** Discussion with prior work.\n\n**A6:** Thanks for the suggestion. \n- [A] proposed semantic guarantees for DP mechanisms against training data reconstruction and two privacy accounting methods are evaluated based on their guarantees. The difference between [A] and our work lies in: \n1. First, [A] discusses the model inversion attack, which is a weaker data reconstruction attack without gradient information compared to gradient inversion. \n2. Second, [A] considers the bound of reconstruction error under the differential privacy framework, while our metric aims to measure the privacy risk under a general perturbation on the sample or gradient. Our work can evaluate the privacy risk under more defense mechanisms like gradient pruning or data perturbation.\n- [B] proposes an upper bound of the probability of the success of a reconstruction attack and shows that DP parameters are not sufficient to estimate the success of the attack. The difference between [B] and our work lies in:\n    - We provide a different risk measurement against [B]. [B] captures the privacy risk via the success probability of the reconstruction attack, while our work directly evaluates the reconstruction error under the worst case. \n\nWe will include this discussion in the revision.\n\n**Q1:** Larger variance of CIFAR10 in Fig.2 is because .\n\n**A1:** Yes. We agree with you. Since CIFAR10 is more complex than MNIST, the variance of CIFAR10 is larger than that of MNIST.\n\n**Q2:** Noise variance in Fig1 and 2\n\n**A2:** It is $10^{-3}$, $10^{-4}$, $10^{-5}$, $10^{-6}$ from light to dark colors.\n\n\n[A] Guo, Chuan, et al. \"Bounding training data reconstruction in private (deep) learning.\" ICML, 2022\n\n[B] Hayes Jamie et al. \"Bounding Training Data Reconstruction in DP-SGD.\" arXiv 2023\n\n[C] Zhu, Ligeng et al. \"Deep leakage from gradients.\" NeurIPS 2019\n\n[D] Huang, Yangsibo, et al. \"Evaluating gradient inversion attacks and defenses in federated learning.\" NeurIPS 2021\n\n[E] Gao, Wei, et al. \"Privacy-preserving collaborative learning with automatic transformation search.\" CVPR. 2021\n\n[F] Sun, J., et al. \"Provable defense against privacy leakage in federated learning from representation perspective. arXiv 2020"}, {"review_id": "2qbs7086YF", "reviewer": "Reviewer_FFT2", "summary": "The authors propose Inversion Influence Function ($I^2F$), a closed-form lower-bound approximation that estimates the recovery $L_2$-norm caused by gradient perturbation in gradient inversion attacks. Detailed mathematical proof and experiments are provided, with comparisons of privacy vulnerability with regard to data, model, perturbation, and attack methods.", "questions": "- What is the perturbation scale used in Fig. (4)?\n- In Fig. (5), are gradient perturbations sampled from Gaussian noise, or do the follow the direction of the eigenvector with the smallest eigenvalue?", "limitations": "The proposed $I^2F$ relies on 5 assumptions that may not hold true. Out of these assumptions, 3.2  is satisfied by common loss functions, 3.3 is bypassed by \"Extension to singular Jacobians\". However, other assumptions, as well as the variables they introduce ($\\mu_J$ and $\\mu_L$), have not been well discussed.\n\n> After rebuttal\n\nWe found the proposed method partially overlaps with [the Fisher Information Loss](https://arxiv.org/pdf/2102.11673.pdf), which weakens its technical novelty. However, there are still differences between the two works in terms of the lower bounds and algorithms they arrive at. I agree with AC that this work could be accepted conditioned on that a more thorough discussion of prior works is included in its final version.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The theorem is supported with detailed math proof and experiment results.\n- $I^2F$ gives a good approximation of privacy vulnerability with regard to data, model, perturbation, and attack methods. Most importantly, one can use it to find a theoretically optimal direction to perturb the gradient in federated learning.", "weaknesses": "- In Lemma B.3. (appendix), $\\nabla_x L_I(x;g) = \\nabla_x \\|\\| \\nabla_\\theta L(x, \\theta) - g \\|\\|^2 = 2 \\nabla_x \\nabla_\\theta L(x, \\theta) (\\nabla_\\theta L(x, \\theta) - g)$, the factor $2$ is missing.\n- For figure 7 in the appendix, there's no explanation of what different lines mean, and it makes no sense to compare the running time of power iteration to that of inversion attack.\n- There should be quoted conclusions/studies on the validity of assumptions, especially the actual value of $\\mu_L$ and $\\mu_J$. If they just exist but take very large values, Eq. (7) will be nonsense.\n- $I_{lb}$ in Fig. (1) is an approximation of Eq. (4) and is different from  $I_{lb}$ defined in Eq. (5), and the computation speedup described in \"Efficient Evaluation\" is not used (only the convergence speed is evaluated).\n- $L_2$-norm is used in almost all math equations, but the experiments use RMSE/MSE, which makes it hard to tell how tight the approximated lower bound is.\n- In Sec 5.3, there exist both $\\sigma(\\theta^Tx)-1$ and $\\sigma(\\theta^Tx)-b$, which looks like an incomplete replacement.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We really appreciate the affirmation of our contribution.\n\n**W1:** In Lemma B.3, factor 2 is missing\n\n**A1:** Thanks for pointing this out. We will revise it accordingly.\n\n**W2:** The explanation of Fig.7 and why comparing the convergence of power iteration and the attack.\n\n**A2:**\n- **What do different lines mean in Fig.7:**\n    - In Fig.7(a), different lines mean the convergence of power iteration under different random seeds.\n    - In Fig.7(b), different lines mean the convergence of the inversion loss under different learning rates (1, 0.5, 0.1, 0.05, 0.01). For each line in (b), it is repeated with 5 different random seeds.\n- **Why compare the running time of power iteration with the attack:** \n    - Conducting the inversion attack can be time-consuming and computationally intensive. **I2F is an attack-free analysis tool that can be used to evaluate the privacy risk given a sample and a perturbation. The efficiency gap between calculating I2F and conducting the inversion attack is a critical metric of the utility of I2F in practice.** Since we use $\\frac{||J\\delta||}{\\lambda_{\\max}(JJ^T)}$ to measure worst-case privacy risk, the computation cost exists in the numerator and denominator, respectively.\n    - The cost of the numerator can be calculated with a Jacobian-vector product, which is equivalent to two times of gradient computation with a vector production. So the numerator can be calculated efficiently with frameworks like PyTorch and its cost is constant for a given model and dataset.\n    - Thus, **the main cost comes from the denominator, which uses power iteration to calculate the maximum eigenvalue of $JJ^T$**. In Fig.7, we show the convergence of power iteration is about 20 times faster than that of the inversion loss, which means that our metric can evaluate the privacy risk accurately and more efficiently.\n\n**W3:** The validity of assumptions, especially the actual values of u_L and u_J. If they are large, then Eq7 makes no sense.\n\n**A3:** Thanks for the comments. \n1. Assumption 3.1 is the assumption of a perfect attack, which is considered the worst-case privacy risk as we claim in lines 133-135. Worst-case risk measurement is currently used in many other areas, such as differential privacy [C]. As gradient inversion attacks are evolving stronger over time, we believe this assumption is reasonable and makes our measurement useful in the future.\n2. Assumption 3.2 is satisfied with the common loss function like cross-entropy loss and was used in the literature [A, B].\n3. For assumption 3.3, first, we agree that “this assumption is bypassed by the ‘Extension to singular Jacobians’” as shown in the limitations by the reviewer. Moreover, we show in Fig.1 and 2 that the lower bound of $I_{lb}$ is a good estimator of I2F. Thus we can directly use $I_{lb}$ to estimate worst-case privacy risk even when $JJ^T$ is singular.\n4. Assumptions 3.4 and 3.5 is not necessary for I2F because I2F only depends on assumptions 3.1-3.3 (lines 151-152). Assumptions 3.4 and 3.5 are only used to provide a theoretical validation of I2F when the noise $\\delta$ is not infinitesimal. We also calculate on LeNet the values of $\\mu_J$ and $\\mu_L$ in assumptions 3.4-3.5. **For CIFAR10, $\\mu_L=0.5014$ and $\\mu_J=1.7\\times10^{-13}$. For MNIST, $\\mu_L=0.7291$ and $\\mu_J=3.7\\times10^{-13}$.** These values are not so large that they are reasonable in practice.\n\n**W4:** I_{lb} in Fig1 is approx of Eq4, and is different from I_{lb} in Eq5. Computation speedup in “Efficient Evaluation” is not used.\n**A4:**\n- **$I_{lb}$ in Fig.1 is different from that in Eq.5:**\n    - We apologize for the confusion. The x-axis in Fig.1, i.e. *$I_{lb}$ (matrix norm)*, is calculated as defined in Eq.5. The matrix norm here is defined as $\\|A\\| = \\sup_{x\\ne0} (\\|Ax\\|/\\|x\\|)$ in line 122. The y-axis in Fig.1 is calculated as Eq.4, which is $(JJ^T)^{-1}J\\delta$. We will clarify to avoid such confusion in the revision.\n- **The computation speedup in “Efficient Evaluation” is not used:**\nThree computation speedup has been proposed in “Efficient Evaluation”. \n    - Speedup (1) of efficient evaluation of $J\\delta$ is used in the whole paper wherever we need to compute the Jacobian-vector product.\n    - Speedup (2), the efficient matrix inversion, is used in Fig.1, where we need to calculate $I$ (matrix inversion) as $(JJ^T)^{-1}J\\delta$ in Eq.4. In Fig1., the results indicate that the lower bound $I_{lb}$ is a good estimator of $I$, so we use $I_{lb}$ instead in the following experiments.\n    - Speedup (3), the efficient evaluation of the Jacobian norm, is used in the whole paper wherever we need to compute the norm of the Jacobian.\n\n\n**W5:** The L2-norm is used in all math equations, but the experiments use MSE/RMSE, which makes it hard to tell how tight the approximated lower bound is.\n\n**A5:** We apologize for the confusion. There is a typo in Eq.1, in which the definition of recovery MSE should be the **root** of recovery MSE as $||x_0-G_r(g_0+\\delta)||$. Also, please kindly check the caption of Fig.1 where RMSE is short for **root of mean square error** instead of *recovery MSE*. We will revise Eq.1 to make it more clear.\n\n**W6:** In Sec5.3, both 1 and b exist.\n\n**A6:** Thanks for pointing this out. There is a typo and it should be $\\sigma(\\theta^{T}x)-b$ instead of $\\sigma(\\theta^{T}x)-1$. We will fix this in the revision.\n\n**Q1:** What is the perturbation scale used in Fig4\n\n**A1:** In Fig.4, we use the eigenvector as the perturbation so the scale is 1.\n\n**Q2:** In Fig5, is the perturbation Gaussian noise or eigenvector?\n\n**A2:** The perturbation is sampled from Gaussian distribution as in the figure caption.\n\n[A] Guo, Chuan, et al. \"Bounding training data reconstruction in private (deep) learning.\" ICML, 2022.\n\n[B] Hannun, Awni, Chuan Guo, and Laurens van der Maaten. \"Measuring data leakage in machine-learning models with Fisher information.\" UAI, 2021.\n\n[C] Dwork, Cynthia. \"Differential privacy.\" Springer Berlin Heidelberg, 2006.\n"}, {"review_id": "y6hjoqMc3X", "reviewer": "Reviewer_VRVE", "summary": "This paper proposes inverse influence function (IIF), the indicator of how reconstructed input (from gradient inversion attack) changes with respect to a gradient change. This function can be simply formulated using Jacobian and gradient change. The correlation between the proposed measure and reconstruction quality is demonstrated both theoretically and empirically. Also, IIF can predict the vulnerability of gaussian perturbation (probably for privacy protection) based on Jacobian (or sample). Using this fact, the sample whose Jacobian has larger maximum eigenvalue is considered to be still unsafe from perturbation-based protection. This fact is also empirically demonstrated on MNIST. Various model initialization schemes are also analyzed using eigenvalues of Jacobian.", "questions": "Please see the weaknesses for basic questions.\nAn another question: what is the conclusion from several analyses on model initialization or perturbation-based protection using IIF?", "limitations": "Limitation is included in the submission.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is easy to follow.\n2. The formulation of IIF is novel and interesting.\n3. The correlation between the proposed measure and reconstruction quality is empirically demonstrated on several datasets\n4. Both theoretical and experimental analyses on Gaussian perturbation based privacy protection using IIF is novel and important.", "weaknesses": "1. There is no experimental result on large-scale dataset and large DNNs.\n2. The worst-case assumption can be very strong in some cases when optimal point from the attack does not reach ground truth.\n3. The theory is strongly based on L2 distance between gradients. How about GS attack case, which is based on cosine-similarity loss?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We really appreciate the valuable comments from the reviewer. We are glad to address the concerns.\n\n**W1:** No experiments on large DNNs and large datasets\n\n**A1:** Thanks for the suggestion. In Fig.5 of the attached PDF, we evaluate our metric on the large model (ResNet152) and large dataset (ImageNet). \n1. For larger models, the MSE is no longer a good metric for the recovery evaluation. Even if state-of-the-art attacks are used and the recovered image is visually similar to the original image in Fig.5(b), the two images are measured to be different by MSE, due to the visual shift: The dog head is shifted toward the left side. To capture such shifted similarity, we use LPIPS [C] instead, which measures the semantic distance between two images instead of the pixel-to-pixel distance like MSE.\n2. Fig.5(a) shows that I2F is correlated to LPIPS using large models and image scales. This implies that **I2F is a good estimator of recovery similarity**.\n\n**W2:** The worst-case assumption can be very strong in some cases when the optimal point from the attack does not reach ground truth.\n\n**A2:** Thanks for the comments. \n- It is easy to see that the ground truth (the private image) is one optimal solution for the attack objective. The problem may be if the optimization algorithm can converge to the desired optimal solution. If this is the problem, we believe the attack can evolve to be much stronger in the near future. We have already seen many recent attack approaches that are getting increasingly powerful [D, E]. Thus, it is important to bound the risk in advance before the risk becomes true.\n- **In addition**, worst-case risk estimation is a common practice. For example, differential privacy [A, B] bound the worst-case chance that the sample is identifiable.  \n\n**W3:** Theory is based on L2-norm attack. How about the cos-sim attack?\n\n**A3:** Thanks for this question. **Our metric is also applicable for evaluating cos-sim attacks**. \n- Note that the minimizer of the L2-norm attack is one solution to the cos-sim (GS) attack. That means **the cos-sim attack can be attained by an optimal L2-norm attack which is our assumption**. Therefore, the L2-norm-based theorem is applicable to the cos-sim attack, as well.\n- **Empirically**, we evaluate $I_{lb}$ on GS attack based on cosine-similarity inversion loss as in Fig.2 and Fig.12 of the main body and appendix of our paper, respectively. It shows that **$I_{lb}$ is linearly correlated to the metrics of MSE, which proves the utility of our metric under GS attack**. We will include the discussion in the revision.\n\n**Q1:** What is the conclusion from several analyses on model initialization or perturbation-based protection using IIF?\n\n**A1:**  Thanks for the question.\n1. Firstly, we find that the **unfairness of privacy protection exists in both samples and classes**. Even for the same noise (distribution), some samples or classes may be more vulnerable to leak privacy as shown in **Fig.5**.\n2. Secondly, **the protection effect of the perturbation itself is not fair**, which is implied from the results that the eigenvectors with smaller eigenvalues are more effective on protection as shown in **Fig.4**, even though all the eigenvectors have the same norm. \n3. Thirdly, since a well-trained model is much more difficult to be inverted with gradient, we evaluate the effect of different initializations on inversion and find that **Kaiming and Xavier have less privacy risk than the normal and uniform initializations** as shown in **Fig.6**. We will include a more clear conclusion in the revision.\n\n[A] Dwork, Cynthia. \"Differential privacy.\" International colloquium on automata, languages, and programming. Berlin, Heidelberg: Springer Berlin Heidelberg, 2006.\n\n[B] Abadi, Martin, et al. \"Deep learning with differential privacy.\" ACM SIGSAC. 2016.\n\n[C] Zhang, Richard, et al. \"The unreasonable effectiveness of deep features as a perceptual metric.\" CVPR. 2018.\n\n[D] Haim, Niv, et al. \"Reconstructing training data from trained neural networks.\" NeurIPS (2022)\n\n[E] Kariyappa, Sanjay, et al. \"Cocktail party attack: Breaking aggregation-based privacy in federated learning using independent component analysis.\" ICML, 2023."}, {"review_id": "br5TLuw3ZH", "reviewer": "Reviewer_X287", "summary": "**Key Contribution:** This work proposes a new method for analysis of privacy risk in the deep leakage from gradients attack, which does not rely on assumptions of the model architecture or attack optimization method.\n\n**Approach:** The work defines an “Inverse Influence Function” (I2F) which is able to determine some metric of privacy risk given an applied defense to gradients, irrespective of the model architecture or attack optimization technique. The authors use their framework to identify that the eigenvalues of the matrix $JJ^T$ greatly impacts the MSE of any possible recoveries - in particular, the paper identifies that the smallest eigenvalues can greatly reduce the risks of leakage.\n\n**Evaluation:** The paper evaluates their approach using LeNet and ResNet18, trained on MNIST and CIFAR10. They consider the impacts of different eigenvalues on the predicted MSE in their I2F framework, and find that the most risky samples (i.e. lowest MSE recoveries) tend to be those where the jacobian had large eigenvalues. The paper also evaluates the impacts of different model initialization schemes on recovery MSE.", "questions": "- (40) What does “converging to the optimal attack” mean in this sentence? Do you mean a perfect recovery of the training data? Or that the optimization method used in the attack itself is optimal?\n\n- (145) Only considering MSE recovery leaves out situations where the recovered image is semantically the same as the private image, but differs in individual pixels. Have you considered alternative metrics which are less sensitive to individual pixels e.g. total variation? \n\n\n**Minor nits:**\nThe original attack is referred to as  “Deep Leakage from Gradients” - is there a reason you chose a different terminology in your paper?", "limitations": "The authors sufficiently addressed the potential social impact and lmiitations of their work.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "**Originality:**\nThe paper reframes analysis of DLG attacks in terms of prior work by Koh and Liang. In this way, it is an original framing of the DLG problem, using established techniques. \n\n**Quality:**\nThe paper is of good quality, providing interesting conclusions and analysis of the I2F approach.\n\n**Clarity:**\nThe paper is well-written and clear - the mathematical steps are simple and easy to follow, with intuitive results.\n\n**Significance:**\nThe paper builds a simple yet intuitive mathematical framework to estimate privacy risk in DLG attacks. The key significance here is in the simplicity and efficiency of the approach, which allows it to be easily used in practice.", "weaknesses": "**Originality:** The paper claims that prior work by  Fan et al. (2020) is not generalizable to different architectures, such as convolutional networks. However, in Fan et al. (2020) Appendix A it is established that such layers can be re-written as fully-connected ones. It is unclear if the paper’s claims about the limitations of prior approaches are true. A more in-depth comparison with prior would help establish the novelty of the paper.\n\n**Limited Evaluation:** The scope of the evaluation could be broadened to further strengthen the paper’s conclusions. It would be interesting to see how the estimation of MSE changes with respect to training iterations - prior work (as cited in the paper) demonstrates that trained models are more susceptible to the DLG attack, and hence justifying this empirical finding with the I2F framework would lend more credence to its utility. Moreover, the paper only considers two defenses: gradients perturbation, and mixing of training samples. However, the paper could be stronger if it considered more defenses e.g. gradient pruning, which is a commonly considered defense.\n\n**Limited Applicability to Realistic Scenarios:** The paper cites Huang et al. 2021 and mentions that BN statistics are needed for a realistic inversion. However, it is unclear how the lack of BN statistics impacts the I2F framework’s utility in a realistic scenario.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks a lot for the positive views of the quality and significance of our work! We are glad to address your concerns.\n\n**W1: (Originality)** Fan et al. (2020) can generalize to other architectures, like CNNs.\n\n**A1:** Thanks for the question. \n- Fan et al. (2020) mentioned that their method can be generalized to convolutional layers. We apologize for the mistake and will revise it in the final paper.\n- The main difference between our work and Fan et al. (2020): Fan et al. (2020) provided a good understanding of when the initial input data be determined and propose an **upper bound of the relative error**, which measures the best-case privacy risk considering the weakest attack. However, our work assumes there exists a stronger attack and focuses on the **lower bound of the RMSE**. Thus, what we consider is the worst-case privacy risk. Both the upper and lower bound are important to better understanding towards gradient inversion attack. So we believe **Fan et al. (2020) and our work are both critical to gradient inversion attack**.\n- **Moreover**, other privacy quantifications like differential privacy [E, F] also measure the worst-case privacy risk, so we believe our target following prior work on the worst case is reasonable.\n\n**W2 (Limited Evaluation):** How does I2F change in training? Evaluate I2F on grad pruning.\n\n**A2:** Thanks for the suggestion.\n- **I2F during training:** In Fig.4 of the attached PDF, we evaluate the change of $I_{lb}$ and recovery MSE during model training. It is observed that **$I_{lb}$ increases by epochs, which means a decreasing privacy risk**. This is consistent with the previous existing empirical results that a well-trained model is more difficult to be inverted by gradient than a randomly initialized model [B, C] (as we claimed in lines 290-292).\n- **I2F with gradient pruning:** In Fig.1 of the attached PDF, we evaluate I2F under the gradient pruning defense where I2F is linearly correlated with RMSE. Thus, **I$^2$F is generalizable with gradient pruning defense**.\n\nWe will add this interesting discussion to the final paper. \n\n**W3 (Limited Applicability):** Based on Huang et al. (2021), how does the lack of BN impact the I2F framework’s utility\n\n**A3:** Thanks for the comments. \n- First, we are at a different stand than Huang et al. (2021). Huang et al. aim to strengthen the attack by leveraging the BN statistics which was shown to be more effective than attacks without BN statistics. In other words, **lacking BN statistics as regularization may weaken the attacks**.\n- In contrast, we aim to estimate the risks from potential gradient inversion attacks from the victim's stand. The victim, who provides the gradient, always has access to the full network including BN statistics. **Without BN statistics, we may underestimate the risks and therefore leave our victim at stake**.\n- Unlike Huang et al., BN statistics are not explicitly used for regularizing and improving inversion attacks in I2F. With the same I2F formulation, **we can estimate the risks from both vanilla gradient inversion or the one with BN-guided regularization (Huang et al., 2021)**.\n\n**Q1:** What does “converging to the optimal attack” mean (line 40)\n\n**A1:** We assume the attack is perfect given the exact gradient of a sample.\n- Note that the sample itself is a solution for Eq(1). If the solution of the inversion attack is unique, then it is equivalent to saying the sample is the unique optimal solution for solving Eq (1). Thus, given the exact gradient of a sample, the attack can exactly recover the sample.\n- Even if the solution is not unique, we can still essentially assume the attack can attain the sample in the worst case. \n- When noise exists, it is unlikely to recover the original images, but we assume the attack can recover the image that is closest to the original image. We will revise the paper to make it more clear.\n\n**Q2:** MSE lacks semantic info. Try other metrics that are less sensitive to individual pixels.\n\n**A2:** Thanks for the suggestion. \n- We follow most existing work [B, C, D] to use MSE (or PSNR, which is negatively correlated with MSE) as the default measure but we agree on the insufficiency of MSE for larger and more complex models. \n- In Fig.3 of the attached PDF, we try two metrics that capture more structural and semantic information between images, i.e., SSIM and LPIPS[A]. SSIM measures the structural similarity between two images, and LPIPS measures the semantic distance. We consider RN18 on CIFAR10 and LeNet on MNIST both with GS and DGL attacks. We find that **$I_{lb}$ is linearly correlated with these two metrics and is a good estimator for the structural similarity and semantic distance between the original and recovered images**.\n\n**Minor:** Why use DGL instead of DLG (Deep leakage from gradients)\n\n**A:** We used DGL because that “Deep Gradient Leakage” is more concise than “Deep Leakage from Gradients”. We will revise in final paper to avoid confusion.\n\n[A] Zhang, Richard, et al. \"The unreasonable effectiveness of deep features as a perceptual metric.\" CVPR. 2018.\n\n[B] Balunović, Mislav, et al. \"Bayesian framework for gradient leakage.\" arXiv (2021).\n\n[C] Geiping, Jonas, et al. \"Inverting gradients-how easy is it to break privacy in federated learning?.\" NeurIPS (2020).\n\n[D] Zhu, Ligeng, Zhijian Liu, and Song Han. \"Deep leakage from gradients.\" NeurIPS (2019).\n\n[E] Dwork, Cynthia. \"Differential privacy.\" International colloquium on automata, languages, and programming. Berlin, Heidelberg: Springer Berlin Heidelberg, 2006.\n\n[F] Abadi, Martin, et al. \"Deep learning with differential privacy.\" ACM SIGSAC. 2016.\n"}]

Author rebuttal:
Thanks to all the reviewers for their patient reading and valuable comments. We are trying our best to address the concerns of all the reviewers. Here we attach a PDF for more empirical results.

Meta-review: This paper proposes a novel influence function-based analysis of gradient perturbation. Using this analysis, the authors show that top singular vectors of the gradient Jacobian matrix can predict the optimal direction of perturbation to protect against gradient inversion, and reveals interesting insights such that privacy protection is not equal across training samples. Reviewers unanimously agree that the analysis is novel and can be used to aid the design of future inversion attacks and defense mechanisms.

The most major weakness is lack of discussion about prior work. Reviewer BBkr suggested two relevant works on bounding data reconstruction attacks against DP-SGD (Guo et al. https://arxiv.org/abs/2201.12383 and Hayes et al. https://arxiv.org/abs/2302.07225). In fact, gradient inversion is a special case of data reconstruction when the model is trained for a single step from a known intermediate state. Moreover, the analysis of Guo et al. using Fisher information, when specialized to the single step setting, is also based on the gradient Jacobian matrix and should be more thoroughly discussed and compared against.

Ultimately, reviewers agree that the authors' analysis is sufficiently different from prior work to be considered novel, but urge the authors to more carefully position their work in light of prior works, especially that of Guo et al. AC recommends acceptance conditioned on this modification to the manuscript.
Decision: Accept (poster)