Title: Understanding Deep Gradient Leakage via Inversion Influence Functions
Venue: NeurIPS 2023
Accepted: yes
Keywords: Deep Learning

Privacy

Federated Learning

Influence Function
Abstract: Deep Gradient Leakage (DGL) is a highly effective attack that recovers private training images from gradient vectors.
This attack casts significant privacy challenges on distributed learning from clients with sensitive data, where clients are required to share gradients.
  Defending against such attacks requires but lacks an understanding of when and how privacy leakage happens, mostly because of the black-box nature of deep networks.
  In this paper, we propose a novel Inversion Influence Function (I$^2$F) that establishes a closed-form connection between the recovered images and the private gradients by implicitly solving the DGL problem.
  Compared to directly solving DGL, I$^2$F is scalable for analyzing deep networks, requiring only oracle access to gradients and Jacobian-vector products.
  We empirically demonstrate that I$^2$F effectively approximated the DGL generally on different model architectures, datasets, modalities, attack implementations, and perturbation-based defenses.
  With this novel tool, we provide insights into effective gradient perturbation directions, the unfairness of privacy protection, and privacy-preferred model initialization.
  Our codes are provided in https://github.com/illidanlab/inversion-influence-function.

Understanding Deep Gradient Leakage via Inversion Influence Functions

Abstract

Deep Gradient Leakage (DGL) is a highly effective attack that recovers private training images from gradient vectors. This attack casts significant privacy challenges on distributed learning from clients with sensitive data, where clients are required to share gradients. Defending against such attacks requires but lacks an understanding of when and how privacy leakage happens , mostly because of the black-box nature of deep networks. In this paper, we propose a novel Inversion Influence Function (I 2 F) that establishes a closed-form connection between the recovered images and the private gradients by implicitly solving the DGL problem. Compared to directly solving DGL, I 2 F is scalable for analyzing deep networks, requiring only oracle access to gradients and Jacobian-vector products. We empirically demonstrate that I 2 F effectively approximated the DGL generally on different model architectures, datasets, modalities, attack implementations, and perturbation-based defenses. With this novel tool, we provide insights into effective gradient perturbation directions, the unfairness of privacy protection, and privacy-preferred model initialization. Our codes are provided in https://github.com/illidanlab/inversion-influence-function .

1 Introduction

With the growing demands for more data and distributed computation resources, distributed learning gains its popularity in training large models from massive and distributed data sources (McMahan et al., 2017; Gemulla et al., 2011). While most distributed learning requires participating clients to share only gradients or equivalents and provides data confidentiality, adversaries can exploit such information to infer training data. Deep Gradient Leakage (DGL) showed such a real-world risk in distributed learning of deep networks (Zhu et al., 2019; Jin et al., 2021). Only given a private gradient and the corresponding model parameters, DGL reverse-engineers the gradient and recovers the private images with high quality. The astonishing success of the DGL started an intensive race between privacy defenders (Sun et al., 2020; Wei et al., 2021; Gao et al., 2021; Scheliga et al., 2022) and attackers (Geiping et al., 2020; Zhao et al., 2020; Jin et al., 2021; Jeon et al., 2021; Zhu and Blaschko, 2021; Fowl et al., 2022).

A crucial question to be answered behind the adversarial game is when and how DGL happens , understanding of which is fundamental for designing attacks and defenses. Recent advances in

theoretical analysis shed light on the question: Balunovi´ c et al. (2022) unified different attacks into a Bayesian framework, leading to Bayesian optimal adversaries, showing that finding the optimal solution for the DGL problem might be non-trivial and heavily relies on prior knowledge. Pan et al. (2022); Wang et al. (2023b) advanced the understanding of the leakage in a specific class of network architectures, for example, full-connected networks. However, it remains unclear how the privacy leakage happens for a broader range of models, for example, the deep convolutional networks, and there is a pressing need for a model-agnostic analysis.

In this paper, we answer this question by tracing the recovered sample by DGL algorithms back to the private gradient, where the privacy leakage ultimately derives from. To formalize the optimal DGL adversary without practical difficulties in optimization, we ask the counterpart question: what would happen if we slightly change the private gradient?

Answering this question by perturbing the gradient and re-evaluating the attack can be prohibitive due to the difficulty of converging to the optimal attack in a highly non-convex space. Accordingly, we propose a novel Inversion Influence Function (I 2 F) that provides an analytical description of the connection between gradient inversion and perturbation. Compared to directly solving DGL, the analytical solution in I 2 F efficiently scales up for deep networks and requires only oracle access to gradients and Jacobian-vector products. We note that I 2 F shares the same spirit of the influence function (Koh and Liang, 2017; Hampel, 1974) that describes how optimal model parameters would change upon the perturbation of a sample and I 2 F can be considered as an extension of the influence function from model training to gradient inversion.

The proposed I 2 F characterizes leakage through the lens of private gradients and provides a powerful tool and new perspectives to inspect the privacy leakage: (1) First, we find that gradient perturbation is not homogeneous in protecting the privacy and is more effective if it aligns with the Jacobian singular vector with smaller singular values. (2) As the Jacobian hinges on the samples, the variety of their Jacobian structures may result in unfair privacy protection under homogeneous Gaussian noise. (3) We also examine how the initialization of model parameters reshapes the Jocabian and therefore leads to quite distinct privacy risks.

These new insights provide useful tips on how to defend DGL, such as perturbing gradient in specific directions rather than homogeneously, watching the unfairness of protection, and carefully initializing models. We envision such insights could lead to the development of fine-grained privacy protection mechanisms. Overall, our contributions can be summarized as follows. (1) For the first time, we introduce the influence function for analyzing DGL; (2) We show both theoretical and empirical evidence of the effectiveness of the proposed I 2 F which efficiently approximates the DGL in different settings; (3) The tool brings in multiple new insights into when and how privacy leakage happens and provides a tool to gauge improved design of attack and defense in the future.

2 Related Work

Attack. Deep Gradient Leakage (DGL) is the first practical privacy attack on deep networks (Zhu et al., 2019) by only matching gradients of private and synthetic samples. As accurately optimizing the objective could be non-trivial for nonlinear networks, a line of work has been developed to strengthen the attack (Zhao et al., 2020). Geiping et al. (2020) introduce scaling invariance via cosine similarity. Balunovi´ c et al. (2022) summarizes these attacks as a variety of prior choices. The theorem is echoed by empirical studies that show better attacks using advanced image priors (Jeon et al., 2021). It was shown that allowing architecture modification could weigh in higher risks in specific cases (Zhu and Blaschko, 2021; Fowl et al., 2022). Besides the study of gradient leakage in initialized models, recent evidence showed that the well-trained deep network can also leak the private samples (Haim et al., 2022). Yet, such an attack still requires strict conditions to succeed, e.g., small dataset and gradient flow assumptions. Though these attacks were shown to be promising, exactly solving the attack problem for all types of models or data is still time-consuming and can be intractable in reasonable time (Huang et al., 2021).

Defense. Most defense mechanisms introduce a perturbation in the pipeline of gradient computation. 1) Gradient Noise. Since the DGL is centered on reverse engineering gradients, the straightforward defense is to modify the gradient itself. Zhu et al. (2019) investigated a simple defense by Gaussian noising gradients or pruning low-magnitude parameters-based defense. The Gaussian noise was motivated by Differential Privacy Gradient Descent (Dwork, 2006; Abadi et al., 2016), which

guarantees privacy in an aggregated gradient. Except for random perturbation, the perturbation could also be guided by representation sensitivity (Sun et al., 2020) or information bottleneck Scheliga et al. (2022). 2) Sample Noise. The intuition of sample noise is to change the source of the private gradient. For example, Gao et al. (2021) proposed to seek transformations such that the transformed images are hard to be recovered. Without extensive searching, simply using strong random data augmentation, e.g., MixUp (Zhang et al., 2017), can also help defense (Huang et al., 2021). If public data are available, hiding private information inside public samples is also an alternative solution (Huang et al., 2020). Though the defenses showed considerate protection against attacks in many empirical studies, the security still conceptually relied on the expected risks of a non-trivial number of evaluations. Explanation of how well a single-shot protection, for example, noising the gradient, is essential yet remains unclear.

Understanding DGL. There are many recent efforts in understanding when noise-based mechanisms help privacy protection. Though Differential Privacy (DP) provides a theoretical guarantee for general privacy, it describes the worst-case protection and does not utilize the behavior of DGL, motivating a line of studies to understand when and how the DGL happens. Sun et al. (2020) connected the privacy leakage to the representation, which aligns with later work on the importance of representation statistics in batch-normalization layers (Hatamizadeh et al., 2021; Huang et al., 2021). Balunovi´ c et al. (2022) unified different attacks in a Bayesian framework where existing attacks differ mainly on the prior probability. Chen and Campbell (2021) investigated the model structure to identify the key reasons for gradient leakage. Hatamizadeh et al. (2021); Huang et al. (2021) evaluated the DGL in practical settings and showed that BN statistics is critical for inversion. There are efforts advancing theoretical understandings of DGL. Pan et al. (2022) studied the security boundary of ReLU activation functions. In a specific class of fully-connected networks, Wang et al. (2023b) showed that a single gradient query at randomized model parameters like DP protection can be used to reconstruct the private sample. Hayes et al. (2023); Fan et al. (2020) proposed an upper bound of the success probability of the data reconstruction attack and the relative recovery error, respectively, which is considered as the least privacy risk. Instead, we consider the worst-case privacy risk.

Though many useful insights are provided, existing approaches typically focus on specific model architectures. This work provides a new tool for analyzing the DGL without assumptions about the model architectures or attacking optimization. Independent of the model architectures, our main observation is that the Jacobian, the joint derivative of loss w.r.t. input and parameters, is the crux.

3 Inversion Influence Function

In this section, we propose a new tool to unravel the black box of DGL. We consider a general loss function L ( x, θ ) defined on the joint space of data X and parameter Θ . For simplicity, we assume x is an image with oracle supervision if considering supervised learning. Let x 0 be the private data and g 0 ≜ ∇ θ L ( x 0 , θ ) be its corresponding gradient which is treated as a constant. An inversion attack aims at reverse engineering g 0 to obtain x 0 . Deep Gradient Leakage (DGL) attacks by directly searching for a synthetic sample x to reproduce the private gradient g by ∇ θ L ( x, θ ) (Zhu et al., 2019). Formally, the DGL is defined as a mapping from a gradient g to a synthetic sample x ∗ g :

̸

where L I is the inversion loss matching two gradients, ∥·∥ denotes the ℓ 2 -norm either for a vector or a matrix. The L 2 -norm of a matrix A is defined by the induced norm ∥ A ∥ = sup x =0 ( ∥ Ax ∥ / ∥ x ∥ ) . The Euclidean distance induced by ℓ 2 -norm can be replaced by other distances, e.g., cosine similarity (Geiping et al., 2020), which is out of our scope, though.

Our goal is to understand how the gradient of a deep network at input x 0 encloses the information of x 0 . For a linear model, e.g., L ( x, θ ) = x ⊤ θ , the gradient is x and directly exposes the direction of the private sample. However, the task gets complicated when a deep neural network is used and L ( x, θ ) is a nonlinear loss function. The complicated functional makes exactly solving the minimization in Eq. (1) highly non-trivial and requires intensive engineering in hyperparameters and computation (Huang et al., 2021). Therefore, analyses tied to a specific DGL algorithm may not be generalizable. We introduce an assumption of a perfect attacker to achieve a generalizable analysis:

Assumption 3.1. Given a gradient vector g , there is only a unique minimizer for L I ( x ; g ) .

Since ∇ θ L ( x, θ ) is a minimizer for G r ( ∇ θ L ( x, θ )) , Assumption 3.1 implies that G r ( ∇ θ L ( x, θ )) is a perfect inversion attack that recovers x exactly. The assumption of a perfect attacker considers the worst-case. It allows us to develop a concrete analysis, even though such an attack may not always be feasible in practice, for example, in non-linear deep networks. To start, we outline the desired properties for our privacy analysis. (1) Efficiency : Privacy evaluation should be efficient in terms of computation and memory; (2) Proximity : The alternative should provide a good approximation or a lower bound of the risk, at least in the high-risk region; (3) Generality : The evaluation should be general for different models, datasets, and attacks.

3.1 Perturbing the Private Gradient

To figure out the association between the leakage and the gradient g , we formalize a counterfactual: what kind of defense can diminish the leakage? A general noise-based defense can be written as g = ∇ θ L ( x 0 , θ ) + δ where δ is a small perturbation. According to Assumption 3.1, a zero δ is not private. Instead, we are interested in non-zero δ and the recovery error (RE) of the recovered image:

For further analysis, we make two common and essential assumptions as follows.

Assumption 3.2. L ( x, θ ) is twice-differentiable w.r.t. x and θ , i.e., J ≜ ∇ x ∇ θ L ( x, θ ) ∈ R d x × d θ exist, where d x and d θ are the dimensions of x and θ , respectively.

Assumption 3.3. JJ ⊤ is invertible.

We then approximate G r ( g 0 + δ ) by the first-order Taylor expansion:

By the implicit function theorem, we can show that ∂G r ( g 0 ) ∂g 0 = ( JJ ⊤ ) -1 J (proof in Appendix B.1). Thus, for a small perturbation δ , we can approximate the privacy leakage through DGL by

The I 2 F includes a matrix inversion, computing which may be expensive and unstable for singular matrixes. Thus, we use a tractable lower bound of I 2 F as:

where λ max ( A ) denotes the maximal eigenvalues of a matrix A . Computing the maximal eigenvalue is usually much cheaper than the matrix inversion. We note that the approximation uses Assumption 3.1 and provides a general risk measurement of different attacks.

The lower bound describes how risky the gradient is in the worst case. This follows the similar intuition of Differential Privacy (DP) (Dwork, 2006) to bound the max chance (worst case) of identifying a private sample from statistic aggregation. The proposed lower bound differentiates the DP worst case in that it leverages the problem structure of gradient inversion and therefore characterizes the DGL risk more precisely.

Efficient Evaluation. The evaluation of I or its lower bound I lb is non-trivial due to three parts: computation of the Jacobian, the matrix inversion, and eigenvalue. The former two computations can be efficiently evaluated using established techniques, for example, (Koh and Liang, 2017). 1) Efficient evaluation of Jδ . Instead of computing the dense Jacobian, an efficient alternative is to leverage the Jacobian-vector product, which is similar to the Hessian-vector product. We can efficiently evaluate the product of Jacobian and δ by rewriting the product as Jδ = ∇ x ( ∇ ⊤ θ L ( x 0 , θ ) δ ) . Since ∇ ⊤ θ L ( x 0 , θ ) δ is a scalar, the second derivative w.r.t. x can be efficiently evaluated by autograd tools, e.g., PyTorch. The computation complexity is equivalent to two times of gradient evaluation in addition to one vector production. 2) Efficient matrix inversion. We can compute b ≜ ( JJ ⊤ ) -1 Jδ by solving min b 1 2 ∥ ∥ δ -J ⊤ b ∥ ∥ 2 , whose gradient computation only contains two Jacobian-vector products. There are other alternative techniques, for example, Neumann series or stochastic approximation, as discussed in (Koh and Liang, 2017). For brevity, we discuss these alternatives in Appendix A.

In this paper, we use the least square trick and solve the minimization by gradient descent directly. 3) Efficient evaluation of privacy with the norm of Jacobian. Because that ∥ J ∥ = √ λ max ( JJ ⊤ ) , we need to compute the maximum eigenvalue of JJ ⊤ , which can be done efficiently by power iteration and the trick of Jacobian-vector products.

Complexity. Suppose the computation complexity for evaluating gradient is a constant in the order O ( d ) where d is the maximum between the dimension of θ and x . Then evaluating the Jacobianvector product is of complexity O ( d ) . By the least square trick with T iterations, we can compute I in O ( dT ) time. The lower bound I lb replaces the computation of matrix inversion with computation on the max eigenvalue. By T ′ -step power iteration, the complexity is O ( d ( T + T ′ )) .

Case study on Gaussian perturbation. Gaussian noise is induced from DP-SGD (Abadi et al., 2016) where gradients are noised to avoid privacy leakage. Assuming the perturbation follows a standard Gaussian distribution, we can compute the expectation as follows:

where λ i ( J ⊤ J ) is the i -th eigenvalue of J ⊤ J . The above illustrates the critical role of the Jacobian eigenvalues. For a perturbation to offer a good defense and yield a large recovery error, the Jacobian should have at least one small absolute eigenvalue. In other words, the protection is great when the Jacobian is nearly singular or rank deficient. The intuition for the insight is that a large eigenvalue indicates the high sensitivity of gradients on the sample, and therefore an inaccurate synthesized sample will be quickly gauged by the increased inversion loss.

3.2 Theoretic Validation

The proposed I 2 F assumes that the perturbation is small enough for an accurate approximation, and yet it is interesting to know how good the approximation is for larger δ . To study the scenario, we provide a lower bound of the inversion error on a non-infinitesimal δ , using weak Lipschitz assumptions for any two samples x and x ′ .

Theorem 3.1. If Assumption 3.4 and 3.5 hold, then the recovery error satisfies:

The proof is available in Appendix B.2. Similar to I 2 F, the inversion error could be lower bound using the Jacobian matrix. Eq. (7) matches our lower bound I lb in Eq. (5) with an extra term of gradients. The bound implies that our approximation could be effective as a lower bound even for larger δ . Besides, the lower bound provide additional information when δ is large: when ∥ δ ∥ gets infinitely large, the bound converges to Ω( ∥ Jδ ∥ / ( µ J ∥ δ ∥ )) , which does not scale up to infinity. The intuition is that the x is not necessarily unbounded for matching an unbounded 'gradient'.

4 Empirical Validation and Extensions

We empirically show that our metric is general for different attacks with different models on different datasets from different modalities.

Setup. We evaluate our metric on two image-classification datasets: MNIST (LeCun, 1998) and CIFAR10 (Krizhevsky et al., 2009). We use a linear model and a non-convex deep model ResNet18 (He et al., 2015a) (RN18) trained with cross-entropy loss. We evaluate our metric on two popular attacks: the DGL attack (Zhu et al., 2019) and the GS attack (Geiping et al., 2020). DGL attack minimizes the L 2 distance between the synthesized and ground-truth gradient, while the GS attack maximizes the cosine similarity between the synthesized and ground-truth gradient. We use the Adam optimizer with a learning rate of 0.1 to optimize the dummy data. Per attacking iteration, the dummy data is projected into the range of the ground-truth data [0 , 1] with the GS attack. When the DGL attack

10

1

10 0

lb

(matrix norm)

10

1

10 0

(matrix inversion)

10

3

10

2

10

1

10 0

nom

(nominator only)

10

1

10 0

(matrix inversion)

(a) Compare different approximation

10

1

10 0

lb

(matrix norm)

10

1

10 0

RMSE

8×10 4

9×10 4

nom

(nominator only)

10

3

10

2

10

1

10 0

RMSE

(b) Compare I

2

F to RMSE

10

2

10

1

10 0

lb

10 0

RMSE

10

2

10

1

10 0

lb

10 0

RMSE

10

1

10 0

lb

10

1

10 0

RMSE

10

2

10

1

10 0

lb

10

3

10

2

10

1

10 0

RMSE

(a) CIFAR10, DGL, RN18

(b) CIFAR10, GS, RN18

(c) MNIST, GS, RN18

(d) MNIST, GS, Linear

cannot effectively recover the data, e.g., on noised linear gradients, we only conduct the GS attack to show the highest risk. We use the root of MSE (RMSE) to measure the difference between the ground-truth and recovered images. All the experiments are conducted on one NVIDIA RTX A5000 GPU with the PyTorch framework. More details and extended results are attached in Appendix C.

Extension to singular Jacobians. We consider the JJ ⊤ be singular, which often happens with deep neural networks where many eigenvalues are much smaller than the maximum. In this case, the inversion of JJ ⊤ does not exist or could result in unbounded outputs. To avoid the numerical instability, we use ∥ ∥ ( JJ ⊤ + ϵI ) -1 Jδ ∥ ∥ with a constant ϵ . In our experiments, the ϵ is treated as a hyperparameter and will be tuned on samples to best fit the linear relation between RMSE and I .

Batch data. In practice, an attacker may only be able to carry out batch gradient inverting, ∥ ∥ 1 n ∑ n i ∇ θ L ( x i , θ ) -1 n ∑ n i g i ∥ ∥ , which can be upper bounded by 1 n ∑ n i ∥∇ θ L ( x i , θ ) -g i ∥ . Such decomposition can be implemented following (Wen et al., 2022), which modifies the parameters of the final layer to reduce the averaged gradient to an update of a single sample. Thus, we only consider the more severe privacy leakage by individual samples instead of a batch of data.

How well are the influence approximations? In Fig. 1(a), we compare the two approximations of I 2 F. I lb relaxes the matrix inversion in I to the reciprocal of ∥ ∥ JJ ⊤ ∥ ∥ . I nom further drops the denominator and only keep the nominator ∥ Jδ ∥ . The two figures show that I lb serves as a tight approximation to the real influence while I nom is much smaller than the desired value. In Fig. 1(b), we see that the relaxation I lb is also a good approximation or lower bound for the RMSE.

Validation on different models, datasets, and attacks. In Fig. 2, we show that I 2 F can work approximately well in different cases though the approximation accuracy varies by different cases. (1) Attack: In Figs. 2(a) and 2(b), we compare DGL to the Gradient Similarly (GS) (Geiping et al., 2020) that matches gradient using cosine similarity instead of the L 2 distance. We see that different attacks do not differ significantly. (2) Dataset: As shown in Figs. 2(b) and 2(c), different datasets could significantly differ regarding attack effectiveness. The relatively high RMSE implies that CIFAR10 tends to be harder to attack due to more complicated features than MNIST. Our metric suggests that there may exist a stronger inversion attack causing even lower RMSE. (3) Model:

10

5

10

3

10

1

10 0

lb

0.3

0.4

0.5

0.6

LPIPS

(a)

I 2 F=5x10

6

LPIPS=0.24

I 2 F=4x10

3

LPIPS=0.42

I 2 F=1x10

2

LPIPS=0.48

I 2 F=1.1

LPIPS=0.66

(b)

Comparing Figs. 2(c) and 2(d), the linear model presents a better correlation than the ResNet18 (RN18). Because the linear model is more likely to be convex, the first-order Taylor expansion in Eq. (3) can approximate its RMSE than one of the deep networks.

Results on Large Models and Datasets. We also evaluate our I 2 F metric on ResNet152 with ImageNet. For larger models, the RMSE is no longer a good metric for the recovery evaluation. Even if state-of-the-art attacks are used and the recovered image is visually similar to the original image in Fig. 3(a), the two images are measured to be different by RMSE, due to the visual shift: The dog head is shifted toward the left side. To capture such shifted similarity, we use LPIPS (Zhang et al., 2018) instead, which measures the semantic distance between two images instead of the pixel-to-pixel distance like RMSE. Fig. 3(a) shows that I2F is correlated to LPIPS using large models and image scales. This implies that I2F is a good estimator of recovery similarity. In Fig. 3(b), original images with a lower I2F also have a smaller LPIPS, which means a better reconstruction. Recovered images on the right (the German shepherd and the panda) cannot be recognized while those on the left (the owl and the beagle) still have enough semantic information for humans to recognize.

Results on Language Models and Datasets. Since the input space of language models is not continuous like images, current attacks on images cannot be directly transferred to text. Besides, continuously optimized tokens or embeddings should be projected back to the discrete text, which induces a gap between the optimal and the original text. Thus, the investigation of privacy risks in text is an interesting yet non-trivial problem. We evaluate the proposed I 2 F metric on BERT (Devlin et al., 2018) and GPT-2 (Radford et al., 2019) with TAG attack (Deng et al., 2021), which minimizes L 2 and L 1 distance between gradients of the original and recovered images. We use ROUGE-L (Lin, 2004) and Google BLEU (Wu et al., 2016) to measure the semantic similarity between the original text and the recovered text. More experimental details and results with ROUGE-1 and feature MSE are provided in C.2. The results are presented in Fig. 4. A darker color indicates a larger noise variance. Since the ROUGE-L and Google BLEU measure the semantic similarity of the text pair while our metric estimates the difference, two semantic metrics are negatively correlated to I lb . The results demonstrate the effectiveness of our metric in evaluating the privacy risks of discrete data.

5 When Does Privacy Leakage Happen?

Guided by the formulation of I 2 F, we provide insights on when privacy leakage happens based on the proposed metric. Our insights are rooted in the properties of Jacobian. Since we want to consider the whole property of the Jacobian matrix, we choose a shallow convolutional neural network LeNet (LeCun et al., 1998) to trade off the utility and efficiency of experiments by default.

5.1 Perturbations Directions Are Not Equivalent

Implication on choosing δ . Eq. (4) clearly implies that the perturbation is not equal in different directions. Decomposing J = U Σ V ⊤ using Singular Value Decomposition (SVD), we obtain I ( δ ; x 0 ) = ∥ ∥ U Σ -1 V ⊤ δ ∥ ∥ . Thus, δ tends to yield a larger I 2 F value if it aligns with the directions of small eigenvalues of JJ ⊤ .

10

8

10

6

10

4

10

2

lb

10

1

3×10 1

6×10 1

ROUGE-L

(a) ROUGE-L

10

8

10

6

10

4

10

2

lb

10

1

6×10 1

10

2

Google BLEU

(b) Google BLEU

10

8

10

6

10

4

10

2

lb

10

1

3×10 1

6×10 1

ROUGE-L

(c) ROUGE-L

10

8

10

6

10

4

10

2

lb

10

1

6×10 1

10

2

Google BLEU

(d) Google BLEU

10

2

10

1

10 0

0.0

0.2

0.4

0.6

0.8

1.0

1.2

MSE

(a) CIFAR10

10

2

10

1

10 0

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

MSE

(b) MNIST

=0.014

=0.09

=0.2

=2

=0.0098

=0.096

=0.21

=2.2

Singular values of Jacobian of deep networks. In Fig. 5, we examine the eigenvalues of LeNet on the MNIST and CIFAR10 datasets. On both datasets, there are always a few eigenvalues of JJ ⊤ (around 10 -2 ) that are much smaller than the largest one ( ≥ 1 ). Observing such a large gap between JJ ⊤ eigenvalues, it is natural to ask: how do the maximal/minimal eigenvalues affect the DGL attacks?

Comparing eigenvectors in defending DGL. We consider a special case of perturbation by letting δ be an eigenvector of JJ ⊤ . Then the I 2 F will be 1 /λ where λ is the corresponding eigenvalue. We conjecture 1 /λ could predict the MSE of DGL attacks. To verify the conjecture, we choose 4 eigenvectors with distinct eigenvalues per sample. The results

0.0

0.2

0.4

0.6

0.8

1.0

normalized index

10

2

10

1

10 0

eigenvalue

CIFAR10

MNIST

for the LeNet model are present in Fig. 6. We see that the MSE decreases by λ . For the MNIST dataset, the MSEλ relation is very close to the predicted 1 /λ . Though the curve is biased from the ground truth for CIFAR10, we still can use 1 /λ to lower bound the recovery error. The bias in CIFAR10 is probably due to the hardness of recovering the more complicated patterns than the digit images. The recovered images in Fig. 6 suggest that even with the same perturbation scale, there exist many bad directions for defense. In the worst case, the image can be fully covered. The observation is an alerting message to the community: protection using random noise may leak private information .

5.2 Privacy Protection Could Be Unfair

Our analysis of the Gaussian perturbation in Eq. (6) indicates that the privacy risk hinges on the Jacobian and therefore the sample. We conjecture that the resulting privacy risk will vary significantly due to the sample dependence. In Fig. 7, we conduct fine-grained experiments to study the variance of privacy protection, when Gaussian perturbation on gradients is used for protection. We use a well-trained model which is thought to be less risky than initial models, for example, in (Balunovi´ c et al., 2022). Though the average of MSE implies a reasonable privacy degree as reported in previous literature, the large variance delivers the opposite message that some samples or classes are not that

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

0.00

0.02

0.04

0.06

0.08

0.10

0.12

probability

0

1

2

3

4

5

6

7

8

9

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

MSE

original

recovered

MSE

class

safe. In the sense of samples, many samples are more vulnerable than the average. For the classes, some classes are obviously more secure than others. Thus, when the traditional metric focusing on average is used, it may deliver a fake sense of protection unfairly for specific classes or samples.

To understand which kind of samples could be more vulnerable to attack, we look into the Jacobian and ask when JJ ⊤ will have larger eigenvalues and higher risks in Fig. 7. Consider a poisoning attack, where an attacker aims to maximize the training loss by adding noise δ x to the sample x . Suppose the parameter is updated via θ ′ ( x + δ x ) = θ -∇ θ L ( x + δ x , θ ) . Let J δ = ∇ x ∇ θ L ( x 1 + δ x , θ ) and we can derive the maximal loss amplification on a test sample x 1 when perturbing x as:

As shown by the inequality, the sample with large ∥ J δ ∥ may significantly bias the training after mild perturbation δ x . If Assumption 3.4 holds, then ∥ J ∥ can be associated with ∥ J δ ∥ by | ∥ J ∥ - ∥ J δ ∥ | ≤ µ J . Now we can connect the privacy vulnerability to the data poisoning. With Eq. (6), samples that suffer high privacy risks due to large ∥ J ∥ could also be influential in training and can be easily used to poison training.

5.3 Model Initialization Matters

Since Jocabian leans on the parameters, we hypothesize that the way we initialize a model can also impact privacy. Like previous work (Sun et al., 2020; Balunovi´ c et al., 2022; Zhu et al., 2019), we are interested in the privacy risks when a model is randomly initialized without training. Unlike previous work, we further ask which initialization strategy could favor privacy protection under the same Gaussian perturbation on gradients.

Case Study: One-layer network. A one-layer network with nonlinear activation σ ( · ) is a simple yet sufficient model for analyzing initialization. Let L ( x, θ ) = 1 2 ∥ ∥ σ ( θ ⊤ x ) -b ∥ ∥ 2 , where b is a constant. Denote the product θ ⊤ x as a . The Jacobian is given by:

10 1

10 2

10 3

[

2

]

0.2

0.4

0.8

1.6

MSE

uniform

kaiming

normal

xavier

(a) MNIST

10 1

10 2

10 3

[

2

]

0.4

0.8

1.6

3.2

MSE

uniform

kaiming

normal

xavier

(b) CIFAR10

(1) Apparently, a well-trained θ with ( σ ( θ ⊤ x ) -b ) = 0 will cause J to be a rank-one matrix. Note that the gradient will be zero in such a case and no longer provide any information about the private sample. Consistent with our theoretical results, where the inverse Jacobian makes I infinite, the gradient will be perfectly private at any perturbation. (2) If σ ( θ ⊤ x ) -b is non-zero, the singular values of J will depends on how close E [ σ ( θ ⊤ x )] is to b . If E [ σ ( θ ⊤ x )] approaches b at initialization, then J will be approximately singular, and therefore the initialization enhances privacy preservation.

Comparing initialization. To evaluate the effect of initialization on the inversion, we conduct experiments of the LeNet model on MNIST and CIFAR10 datasets with DGL and GS attacks. We add Gaussian noise to the gradients, which implies an expected MSE should be proportional to E [ I 2 ] = ∑ i λ -1 i as proved in Eq. (6). Here, we evaluate how the initialization will change eigenvalues of JJ ⊤ and thus change the MSE of DGL. Four commonly used initialization techniques are considered here: uniform distribution in range [ -0 . 5 , 0 . 5] , normal distribution with zero mean and variance 0 . 5 , kaiming (He et al., 2015b) and xavier (Glorot and Bengio, 2010) initialization. For each initialization, the same 10 samples are used to conduct the attack three times. The results are shown in Fig. 8. We observe a significant gap between initialization mechanisms. Using uniform initialization cast serious risks of leaking privacy under the same Gaussian defense. Though not as significant as uniform initialization, the normal initialization is riskier than rest two techniques. kaiming and xavier methods can favor convergence in deep learning and here we show that they are also preferred for privacy. A potential reason is that the two methods can better normalize the activations to promote the Jacobian singularity.

In parallel with our work, Wang et al. (2023a) also finds that the initialization is impactful on inversion attacks. Despite their analysis being based on a different metric, the layer-wise variance of the model weights, their work and ours have a consistent conclusion that models initialized with a uniform distribution face higher privacy risks than that with a kaiming distribution

6 Conclusion and Discussion

In this paper, we introduce a novel way to use the influence functions for analyzing Deep Gradient Leakage (DGL). We propose a new and efficient approximation of DGL called the Inversion Influence Function (I 2 F). By utilizing this tool, we gain valuable insights into the occurrence and mechanisms of DGL, which can greatly help the future development of effective defense methods.

Limitations. Our work may be limited by some assumptions and approximations. First, we worked on the worst-case scenario where a strong attack conducts perfect inversion attacks. In practice, such an assumption can be strong, especially for highly complicated deep networks. The gap between existing attacks and perfect attacks sometimes leads to a notable bias. However, we note that recent years witnessed many techniques that significantly improved attacking capability (Geiping et al., 2020; Jeon et al., 2021; Zhao et al., 2020), and our work is valuable to bound the risks when the attacks get even stronger over time. Second, similar to the traditional influence function, I 2 F can be less accurate and suffers from large variance in extremely non-convex loss functions. Advanced linearization techniques (Bae et al., 2022) can be helpful in improving the accuracy of influence. Then extending our analysis to bigger foundation models may bring intriguing insights into the scaling law of privacy.

Future Directions. As the first attempt at influence function in DGL, our method can serve multiple purposes to benefit future research. For example, our metric can be used to efficiently examine the privacy breach before sending gradients to third parties. Since I 2 F provides an efficient evaluation of the MSE, it may be directly optimized in conjunction with the loss of main tasks. Such joint optimization could bring in the explicit trade-off between utility and privacy in time. In comparison, traditional arts like differential privacy are complicated by tuning the privacy parameter for the trade-off. Furthermore, we envision that many techniques can be adopted to further enhance the analysis. For example, unrolling-based analysis leverages the iterative derivatives in the DGL to uncover the effectiveness of gradient perturbations (Pruthi et al., 2020).

Broader Impacts. Data privacy has been a long-term challenge in machine learning. Our work provides a fundamental tool to diagnose privacy breaches in the gradients of deep networks. Understanding when and how privacy leakage happens can essentially help the development of defenses. For example, it can be used for designing stronger attacks, which leads to improved defense mechanisms and ultimately benefit the privacy and security of machine learning.

7 Acknowledgments

This research was supported by the National Science Foundation (IIS-2212174, IIS-1749940), National Institute of Aging (IRF1AG072449), and the Office of Naval Research (N00014- 20-12382).

References

Abadi, M., Chu, A., Goodfellow, I., McMahan, H. B., Mironov, I., Talwar, K., and Zhang, L. (2016). Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security , pages 308-318.

Antunes, R. S., André da Costa, C., Küderle, A., Yari, I. A., and Eskofier, B. (2022). Federated learning for healthcare: Systematic review and architecture proposal. ACM Transactions on Intelligent Systems and Technology (TIST) , 13(4):1-23.

Bae, J., Ng, N., Lo, A., Ghassemi, M., and Grosse, R. B. (2022). If influence functions are the answer, then what is the question? Advances in Neural Information Processing Systems , 35:17953-17967.

Balunovi´ c, M., Dimitrov, D. I., Staab, R., and Vechev, M. (2022). Bayesian framework for gradient leakage. ICLR .

Chen, C. and Campbell, N. D. (2021). Understanding training-data leakage from gradients in neural networks for image classification. arXiv preprint arXiv:2111.10178 .

Dayan, I., Roth, H. R., Zhong, A., Harouni, A., Gentili, A., Abidin, A. Z., Liu, A., Costa, A. B., Wood, B. J., Tsai, C.-S., et al. (2021). Federated learning for predicting clinical outcomes in patients with covid-19. Nature medicine , 27(10):1735-1743.

Deng, J., Wang, Y., Li, J., Shang, C., Liu, H., Rajasekaran, S., and Ding, C. (2021). Tag: Gradient attack on transformer-based language models. arXiv preprint arXiv:2103.06819 .

Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 .

Dwork, C. (2006). Differential privacy. In Automata, Languages and Programming: 33rd International Colloquium, ICALP 2006, Venice, Italy, July 10-14, 2006, Proceedings, Part II 33 , pages 1-12. Springer.

Fan, L., Ng, K. W., Ju, C., Zhang, T., Liu, C., Chan, C. S., and Yang, Q. (2020). Rethinking privacy preserving deep learning: How to evaluate and thwart privacy attacks. Federated Learning: Privacy and Incentive , pages 32-50.

Fowl, L. H., Geiping, J., Czaja, W., Goldblum, M., and Goldstein, T. (2022). Robbing the fed: Directly obtaining private data in federated learning with modified models. In International Conference on Learning Representations .

Gao, W., Guo, S., Zhang, T., Qiu, H., Wen, Y., and Liu, Y. (2021). Privacy-preserving collaborative learning with automatic transformation search. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 114-123.

Geiping, J., Bauermeister, H., Dröge, H., and Moeller, M. (2020). Inverting gradients-how easy is it to break privacy in federated learning? Advances in Neural Information Processing Systems , 33:16937-16947.

Gemulla, R., Nijkamp, E., Haas, P. J., and Sismanis, Y. (2011). Large-scale matrix factorization with distributed stochastic gradient descent. In Proceedings of the 17th ACM SIGKDD international conference on Knowledge discovery and data mining , pages 69-77.

Glorot, X. and Bengio, Y. (2010). Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics , pages 249-256. JMLR Workshop and Conference Proceedings.

Grazzi, R., Franceschi, L., Pontil, M., and Salzo, S. (2020). On the iteration complexity of hypergradient computation. In International Conference on Machine Learning , pages 3748-3758. PMLR.

Guo, C., Karrer, B., Chaudhuri, K., and van der Maaten, L. (2022). Bounding training data reconstruction in private (deep) learning. In International Conference on Machine Learning , pages 8056-8071. PMLR.

Haim, N., Vardi, G., Yehudai, G., Shamir, O., et al. (2022). Reconstructing training data from trained neural networks. In Advances in Neural Information Processing Systems .

Hampel, F. R. (1974). The influence curve and its role in robust estimation. Journal of the american statistical association , 69(346):383-393.

Hannun, A., Guo, C., and van der Maaten, L. (2021). Measuring data leakage in machine-learning models with fisher information. In Uncertainty in Artificial Intelligence , pages 760-770. PMLR.

Hatamizadeh, A., Yin, H., Molchanov, P., Myronenko, A., Li, W., Dogra, P., Feng, A., Flores, M., Kautz, J., Xu, D., et al. (2021). Towards understanding the risks of gradient inversion in federated learning.

Hatamizadeh, A., Yin, H., Roth, H. R., Li, W., Kautz, J., Xu, D., and Molchanov, P. (2022). Gradvit: Gradient inversion of vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10021-10030.

Hayes, J., Mahloujifar, S., and Balle, B. (2023). Bounding training data reconstruction in dp-sgd. arXiv preprint arXiv:2302.07225 .

He, K., Zhang, X., Ren, S., and Sun, J. (2015a). Deep residual learning for image recognition. arxiv 2015. arXiv preprint arXiv:1512.03385 , 14.

He, K., Zhang, X., Ren, S., and Sun, J. (2015b). Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision , pages 1026-1034.

Huang, Y., Gupta, S., Song, Z., Li, K., and Arora, S. (2021). Evaluating gradient inversion attacks and defenses in federated learning. Advances in Neural Information Processing Systems , 34:7232-7241.

Huang, Y., Song, Z., Li, K., and Arora, S. (2020). Instahide: Instance-hiding schemes for private distributed learning. In International conference on machine learning , pages 4507-4518. PMLR.

Jeon, J., Lee, K., Oh, S., Ok, J., et al. (2021). Gradient inversion with generative image prior. In Advances in neural information processing systems .

Jin, X., Chen, P.-Y., Hsu, C.-Y., Yu, C.-M., and Chen, T. (2021). Cafe: Catastrophic data leakage in vertical federated learning. Advances in Neural Information Processing Systems , 34:994-1006.

Kaissis, G., Ziller, A., Passerat-Palmbach, J., Ryffel, T., Usynin, D., Trask, A., Lima Jr, I., Mancuso, J., Jungmann, F., Steinborn, M.-M., et al. (2021). End-to-end privacy preserving deep learning on multi-institutional medical imaging. Nature Machine Intelligence , 3(6):473-484.

Koh, P. W. and Liang, P. (2017). Understanding black-box predictions via influence functions. In International conference on machine learning , pages 1885-1894. PMLR.

Krizhevsky, A., Hinton, G., et al. (2009). Learning multiple layers of features from tiny images.

LeCun, Y. (1998). The mnist database of handwritten digits. http://yann. lecun. com/exdb/mnist/ .

LeCun, Y., Bottou, L., Bengio, Y., and Haffner, P. (1998). Gradient-based learning applied to document recognition. Proceedings of the IEEE , 86(11):2278-2324.

Li, H., Li, C., Wang, J., Yang, A., Ma, Z., Zhang, Z., and Hua, D. (2023). Review on security of federated learning and its application in healthcare. Future Generation Computer Systems , 144:271-290.

Lin, C.-Y. (2004). Rouge: A package for automatic evaluation of summaries. In Text summarization branches out , pages 74-81.

Long, G., Tan, Y., Jiang, J., and Zhang, C. (2020). Federated learning for open banking. In Federated Learning: Privacy and Incentive , pages 240-254. Springer.

McMahan, B., Moore, E., Ramage, D., Hampson, S., and y Arcas, B. A. (2017). Communicationefficient learning of deep networks from decentralized data. In Artificial intelligence and statistics , pages 1273-1282. PMLR.

Merity, S., Xiong, C., Bradbury, J., and Socher, R. (2016). Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843 .

Pan, X., Zhang, M., Yan, Y., Zhu, J., and Yang, Z. (2022). Exploring the security boundary of data reconstruction via neuron exclusivity analysis. In 31st USENIX Security Symposium (USENIX Security 22) , pages 3989-4006.

Pruthi, G., Liu, F., Kale, S., and Sundararajan, M. (2020). Estimating training data influence by tracing gradient descent. Advances in Neural Information Processing Systems , 33:19920-19930.

Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. (2019). Language models are unsupervised multitask learners.

Roth, H. R., Chang, K., Singh, P., Neumark, N., Li, W., Gupta, V., Gupta, S., Qu, L., Ihsani, A., Bizzo, B. C., et al. (2020). Federated learning for breast density classification: A realworld implementation. In Domain Adaptation and Representation Transfer, and Distributed and Collaborative Learning: Second MICCAI Workshop, DART 2020, and First MICCAI Workshop, DCL 2020, Held in Conjunction with MICCAI 2020, Lima, Peru, October 4-8, 2020, Proceedings 2 , pages 181-191. Springer.

Scheliga, D., Mäder, P., and Seeland, M. (2022). Precode-a generic model extension to prevent deep gradient leakage. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 1849-1858.

Sun, J., Li, A., Wang, B., Yang, H., Li, H., and Chen, Y. (2020). Provable defense against privacy leakage in federated learning from representation perspective. arXiv preprint arXiv:2012.06043 .

Wang, F., Hugh, E., and Li, B. (2023a). More than enough is too much: Adaptive defenses against gradient leakage in production federated learning. In INFOCOM .

Wang, Z., Lee, J., and Lei, Q. (2023b). Reconstructing training data from model gradient, provably. In International Conference on Artificial Intelligence and Statistics , pages 6595-6612. PMLR.

Wei, W., Liu, L., Wut, Y., Su, G., and Iyengar, A. (2021). Gradient-leakage resilient federated learning. In 2021 IEEE 41st International Conference on Distributed Computing Systems (ICDCS) , pages 797-807. IEEE.

Wen, Y., Geiping, J., Fowl, L., Goldblum, M., and Goldstein, T. (2022). Fishing for user data in large-batch federated learning via gradient magnification. arXiv preprint arXiv:2202.00580 .

Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., Klingner, J., Shah, A., Johnson, M., Liu, X., Łukasz Kaiser, Gouws, S., Kato, Y., Kudo, T., Kazawa, H., Stevens, K., Kurian, G., Patil, N., Wang, W., Young, C., Smith, J., Riesa, J., Rudnick, A., Vinyals, O., Corrado, G., Hughes, M., and Dean, J. (2016). Google's neural machine translation system: Bridging the gap between human and machine translation.

Xu, J., Glicksberg, B. S., Su, C., Walker, P., Bian, J., and Wang, F. (2021). Federated learning for healthcare informatics. Journal of Healthcare Informatics Research , 5:1-19.

Zhang, H., Cisse, M., Dauphin, Y. N., and Lopez-Paz, D. (2017). mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09412 .

Zhang, H., Hong, J., Dong, F., Drew, S., Xue, L., and Zhou, J. (2023). A privacy-preserving hybrid federated learning framework for financial crime detection. arXiv preprint arXiv:2302.03654 .

Zhang, R., Isola, P., Efros, A. A., Shechtman, E., and Wang, O. (2018). The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 586-595.

Zhao, B., Mopuri, K. R., and Bilen, H. (2020). idlg: Improved deep leakage from gradients. arXiv preprint arXiv:2001.02610 .

Zhu, J. and Blaschko, M. B. (2021). R-gap: Recursive gradient attack on privacy. In International Conference on Learning Representations .

Zhu, L., Liu, Z., and Han, S. (2019). Deep leakage from gradients. Advances in neural information processing systems , 32.

A Method

A.1 Other Efficient Evaluation Techniques

The techniques for efficiently evaluating implicit gradients can be referred to (Koh and Liang, 2017; Grazzi et al., 2020). As computing the inverse second-order derivatives is the most computationintensive operation, we will focus on it. Here, we briefly summarize two supplementary techniques introduced in Section 3.1.

Conjucate gradient. In Section 3.1, we use the trick of least square to compute the ( JJ ⊤ ) -1 Jδ . When JJ ⊤ ≻ 0 , we can solve the least square problem by the conjugate gradient (CG) method, which only needs O ( d ) time to converge. However, the algorithm will be unstable when the matrix JJ ⊤ is ill-conditioned. As observed in Fig. 5, the JJ ⊤ is likely to be ill-conditioned for deep networks. But here we provide this alternative for linear models such that J can be evaluated faster.

Neumann series. We can leverage the Neumann series to compute the matrix inverse. By the Neumann series, we have ( JJ ⊤ ) -1 Jδ = lim t →∞ ∑ t i =0 ( I -JJ ⊤ ) i Jδ . Let s t ≜ ∑ t i =0 ( I -JJ ⊤ ) i Jδ and s 0 ≜ Jδ . Then the computation can be done by iteration s t +1 = ( I -JJ ⊤ ) s t + Jδ which only includes Jacobian-vector products.

B Proofs

B.1 Proof of the Approximation by Implicit Gradients

Here, we provide the proof for ∂G r ( g 0 ) ∂g 0 = ( JJ ⊤ ) -1 J . Recall Eq. (1) as

The stationary condition of the minimization gives

Given a small perturbation ∆ g → 0 on the gradient, we can estimate corresponding perturbation ∆ x ∗ → 0 as a function of ∆ g . Thus, we can approximate ∂x ∗ ∂g by ∂ ∆ x ∗ ∂ ∆ g . Use Taylor expansion to show approximately

where we drop higher-order perturbations. The above derivations can be rigorously proved using the Implicit Function Theorem. Since ∂L I ( x ∗ ; g ) ∂x ∗ = 2( ∇ θ L ( x ∗ , θ ) -g ) ∇ x ∇ θ L ( x ∗ , θ ) , we can derive

and

As x 0 = x ∗ = G r ( g 0 ) and g 0 = ∇ θ L ( x ∗ , θ ) , we can substitute them to obtain

Let g = g 0 . Combine Eqs. (9) to (11) to get

B.2 Proof of Theorem 3.1

Before we prove our main theorem, we prove several essential lemmas as below.

Proof.

Using Assumption 3.4 and 3.5 directly lead to

Proof. By Assumption 3.4, we have

Lemma B.3. The inversion loss L I ( x ; g ) defined satisfies ∥∇ x L I ( x ; g ) -∇ x L I ( x ′ ; g ) ∥ ≤ µ ∥ x -x ′ ∥ where

Proof. Since we can derive

By Lemma B.1 and Lemma B.2, we have

where µ 1 = µ L ∥∇ x ∇ θ L ( x, θ ) ∥ + µ J ∥∇ θ L ( x ′ , θ ) ∥ and µ 2 = µ J ∥ g ∥ . Let µ = 2 µ 1 +2 µ 2 . Then we can get

Theorem B.1 (Restated from Theorem 3.1) . Let x 0 be the private data and g 0 ≜ ∇ θ L ( x 0 , θ ) be its corresponding gradient which is treated as a constant. If Assumption 3.4 and 3.5 hold, then the square root of the recovery RMSE satisfies:

where J = ∇ ∇ L ( x , θ )

Proof. Utilize the stationary condition ∇ x L I ( x ∗ g ; g ) = 0 and Lemma B.3 to obtain

As x 0 is the private sample whose gradient is g 0 ≜ ∇ θ L ( x 0 , θ ) , then we have

Because we can attain

With ∇ θ L ( x ∗ g 0 + δ , θ ) = g 0 + δ , we can obtain

C Experiments

C.1 Experimental Details

Model architectures. The linear model we use is a matrix that maps the input data into a vector. The LeNet model is a convolutional neural network with 4 convolutional layers and 1 fully connected layer. We use the modified version following previous privacy papers Sun et al. (2020), whose detailed structure is in Table 1. ResNet18 is a popular deep convolutional network in computer vision with batch-normalization and residual layers (He et al., 2015a). Cross entropy loss is used as the loss function in all the experiments.

Experimental settings. We conduct two kinds of attacks in our paper: DGL and GS attacks. The learning rate of the two attacks is 0.1 and we use Adam as the optimizer. To consider a more powerful attack, only a single image is reconstructed in each inversion. When inverting LeNet, we uniformly initialize the model parameters in the range of [ -0 . 5 , 0 . 5] as (Sun et al., 2020) to get a stronger attack. When inverting ResNet18, we use the default initialization method in PyTorch and follow Huang et al. (2021) to use BN statistics as an additional regularization term to conduct a stronger attack.

10

8

10

6

10

4

10

2

lb

10

1

10 0

ROUGE-1

(a) ROUGE-1

10

8

10

6

10

4

10

2

lb

10

1

3×10 1

6×10 1

ROUGE-L

(b) ROUGE-L

10

8

10

6

10

4

10

2

lb

10

1

6×10 1

10

2

Google BLEU

(c) Google BLEU

10

8

10

6

10

4

10

2

lb

10 1

4×10 1

feature MSE

(d) Feature MSE

10

8

10

6

10

4

10

2

lb

10

1

10 0

ROUGE-1

(e) ROUGE-1

10

8

10

6

10

4

10

2

lb

10

1

3×10 1

6×10 1

ROUGE-L

(f) ROUGE-L

10

8

10

6

10

4

10

2

lb

10

1

6×10 1

10

2

Google BLEU

(g) Google BLEU

10

8

10

6

10

4

10

2

lb

10 1

4×10 1

feature MSE

(h) Feature MSE

C.2 Empirical Validation on Language Data

We evaluate the proposed I 2 F metric on BERT (Devlin et al., 2018) and GPT-2 (Radford et al., 2019), which are popular language models in natural language processing. We use TAG (Deng et al., 2021), which is an attack on Transformer-based language models based on the L 1 and L 2 distance between the original gradient and the dummy gradient, and follow the code in https: //github.com/JonasGeiping/breaching . We use the default setting in the code and iteratively update the input embedding 12,000 times. We randomly sample 70 sentences from WikiText103 (Merity et al., 2016) as the private text. We use ROUGE-1 , ROUGE-L (Lin, 2004), Google BLEU (Wu et al., 2016) and feature MSE to measure the semantic similarity between the original text and the recovered text. ROUGE-1 measures the overlap of 1 -grams in the original and recovered text, while ROUGE-L measures the length of the longest common subsequence between two sentences. While ROUGE metrics calculate the reconstruction recall, the Google BLEU score uses as the output the smaller value of the precision and recall of the original and recovered text and has a broader range of the overlap of n -grams, where n = 1 , 2 , 3 , 4 . Since the above three metrics are discrete and not consistent with our assumptions, we include a continuous metric, the feature MSE, which measures the distance between the final layer's feature of the original and recovered text.

The results are presented in Fig. 9. A darker color indicates a larger noise variance. For each noise variance, we randomly sample the perturbation from a zero-mean Gaussian distribution 5 times and repeat this for 3 different random seeds. It shows that I 2 F is correlated to these four metrics, which means I 2 F can be used to estimate the privacy risk of text datasets with large language models. The correlation of BERT and GPT-2 between the four metrics has a similar mode. Though ROUGE-1, ROUGE-L and Google BLEU measure the structural similarity of sentences, which consists of discrete tokens, I 2 F presents a clear correlation. For the feature MSE, although I 2 F has a less distinguished correlation with feature MSE, it can still be utilized to estimate the privacy risk.

C.3 Efficiency

To show the efficiency of computing the I 2 F values, we compare it with the GS attack on randomlypicked CIFAR10 images with ResNet18. As the major complexity comes from evaluating the maximal eigenvalue via the power iteration method, we compare the time required for the power iteration against the inversion loss of GS to converge. We notice that the convergence depends on the initialization of the power iteration and the learning rate for DGL. Thus, we repeat power iteration with

1.0

1.5

2.0

2.5

3.0

3.5

4.0

4.5

running time (sec)

11.8

12.0

12.2

12.4

12.6

12.8

max eigenvalue

(a) Power iteration

0

20

40

60

80 100 120 140 160

running time (sec)

0.25

0.30

0.35

0.40

0.45

0.50

0.55

0.60

0.65

inversion loss

(b) Inversion attack

10 0

10 1

10 2

10 3

time (sec)

2.8

1.9

27

30

167

56

100 27

350

500 367

lb

inversion

(a) DGL attack

10 0

10 1

10 2

10 3

time (sec)

3.3

1.2

0.7

0.8

33

117

700233

25

550

467

lb

inversion

(b) GS attack

5 different seeds. For the inversion attack, we evaluate multiple learning rates ( 1 , 0 . 5 , 0 . 1 , 0 . 05 , 0 . 01 ) to show the fastest convergence. Each experiment is repeated 5 times with different random seeds. As shown in Fig. 10, the power iteration method can converge to the maximal eigenvalue within 50 iterations (5 seconds at most). In comparison, the inversion loss demands 3000 more iterations in 150 seconds to fully converge, which is 20 times larger than the power iteration method. Thus, our method can give an accurate and fast approximation of the recovery MSE without the exhaustive whole inversion process.

In Fig. 11, we compare the computation cost of computing I lb with minimizing inversion loss. We show that for almost all the models and datasets we evaluate, the time ratio is larger than 1 , which means it is more efficient to compute I lb than minimize the inversion loss. It indicates that our method is a more efficient way to estimate the privacy risk for most models and datasets, than the empirical inversion attack. Another key point is that, for large models and datasets, such as models larger than RN18 with CIFAR10 or ImageNet, the time ratio is even much larger than 500 . When the time consumption of inversion attacks on these models and datasets is huge (about 16 minutes or even longer), our method significantly reduces the computation overheads for estimating the privacy risks.

(a)

λ

= 3

.

90

(b)

λ

= 0

.

12

(c)

λ

= 0

.

03

(d)

λ

= 0

.

0006

(a)

2

.

60

(b)

2

.

71

(c)

2

.

39

(d)

3

.

97

(e)

0

.

15

(f)

0

.

28

(g)

0

.

32

(h)

0

.

33

C.4 More Visual Results

More images of unequal perturbations. We present in Fig. 12 the recovered CIFAR10 images when perturbing the gradient with eigenvectors with different eigenvalues. When perturbing with eigenvectors with smaller eigenvalues, the recovered images are more noisy, which is consistent with our former observation.

We also present the unequal perturbations with different eigenvectors of ResNet18 on the CIFAR10 dataset in Fig. 13. Even with the same perturbation scale, the eigenvectors with larger eigenvalues provide stronger protection, where the corresponding recovered images are more noisy and lose some semantic information.

More images of unfair privacy protection. We show more results of unfair privacy protection in Fig. 14. The images of digits 5 and 8 can still be recognized by their outlines, while images of digits 7 and 9 are unrecognizable noise.

We also show the unfair privacy protection of ResNet18 on the CIFAR10 dataset in Fig. 15. In this experiment, we also observe a large variance of recovery MSE among samples, indicating sample-wise unfairness. At the class level, we still can find gradients of a few classes to be easily inverted. For example, class 8 has most MSEs lower than the average value.

D I 2 F with Gradient Pruning Defense

We present the relationship between the RMSE and I lb in Fig. 16. The y-axis is RMSE and the x-axis is I lb . It shows I lb can be used to estimate the worst-case privacy risk with gradient pruning defense.

E Comparison of I 2 F with More Metrics

MSE is a pixel-wise distance that lacks semantic and structural information. To evaluate the effectiveness of I 2 F on more metrics, we consider SSIM and LPIPS (Zhang et al., 2018) to measure the

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

S

2

9

0.2

0.3

0.4

0.5

0.6

MSE

0.00

0.02

0.04

0.06

0.08

0.10

0.12

0.14

0.16

probability

0

1

2

3

4

5

6

7

8

9

class

0.2

0.3

0.4

0.5

0.6

MSE

浙

10

3

10

2

10

1

10 0

10 1

2

1

0.2

(a) RN18-CIFAR10-GS

10

3

10

2

10

1

10 0

10 1

2

1

0.2

(b) RN18-CIFAR10-DGL

10

2

10

1

10 0

10 1

10

1

10 0

(c) RN18-MNIST-GS

10

2

10

1

10 0

10 1

10

1

10 0

(d) RN18-MNIST-DGL

10

2

10

1

10 0

10

1

10 0

(e) LeNet-MNIST-GS

10

2

10

1

10 0

10

1

10 0

10 1

(f) LeNet-MNIST-DGL

10

3

10

2

10

1

10 0

2

1

0.6

(g) LeNet-CIFAR10-GS

10

2

10

1

10 0

4

1

0.4

(h) LeNet-CIFAR10-DGL

structural similarity and semantic distance between the original and recovered images, respectively. The relationship between SSIM and LPIPS is shown in Fig. 17. Since I lb aims to lower bound the privacy risk in terms of RMSE, I lb does not have the same scale as SSIM and LPIPS. Nevertheless, I lb also has a positive correlation between SSIM and LPIPS, which implies that it is a good estimator for the structural similarity and semantic distance between the original and recovered images.

10

2

10

1

10 0

10

3

10

2

10

1

10 0

(a) RN18-CIFAR10-GS

10

2

10

1

10 0

10

2

10

1

10 0

(b)

RN18-CIFAR10-

DGL

10

2

10

1

10 0

10

1

10 0

(c) LeNet-MNIST-GS

10

2

10

1

10 0

10

4

10

3

10

2

10

1

10 0

(d) LeNet-MNIST-DGL

10

2

10

1

10 0

0.1

0.02

0.4

(e) RN18-CIFAR10-GS

10

2

10

1

10 0

0.1

0.02

0.4

(f)

RN18-CIFAR10-

10

2

10

1

10 0

0.1

0.2

0.4

0.8

(g) LeNet-MNIST-GS

10

2

10

1

10 0

0.1

0.2

0.4

0.8

(h) LeNet-MNIST-DGL

DGL

F Dynamics of I lb During Training

Previous existing empirical results show that privacy risk decreases by training epochs (Balunovi´ c et al., 2022; Geiping et al., 2020). We evaluate the dynamics of I lb , RMSE with DGL attack and RMSE during training in Fig. 18. It shows that as the training epoch increases, the I lb also has an increasing trend. While the RMSE of RN18 on the CIFAR10 dataset has a similar trend as I lb , that of LeNet on the MNIST dataset has a significant rise at the epoch 60, which is due to the slower learning speed of LeNet than RN18. Moreover, almost for all the epochs, there is a sample with low I lb , which again emphasizes the unfairness in privacy protection.

G The Impact of ϵ on Efficient Matrix Inversion

In Fig. 19, we study the impact of ϵ on efficient matrix inversion proposed in Section 4. We evaluate the impact on the LeNet with the MNIST dataset. The y-axis is the RMSE. I (matrix inversion) is calculated as defined in Eq. (4). I lb (matrix norm) is calculated as defined in Eq. (5). It shows with ϵ ∈ [1 , 10] , I lb is a lower bound of the RMSE. It also shows that I lb is an accurate estimator of I 2 F. Thus, we can directly use I lb in practice to lower bound the privacy risk to avoid fine-tuning the ϵ .

H Discussion

H.1 Validity of Assumption 3.4-3.5

We make assumptions about the Lipschitz continuous Jacobian and gradient in Assumption 3.4 and Assumption 3.5, respectively. These two assumptions are not necessary for I 2 F but are only used to provide a theoretical validation of I 2 F when the noise δ is not infinitesimal. To discuss the validity of these two assumptions in practice, we calculate the value of µ J and µ L of LeNet with two datasets.

0

20

40

60

80

10

3

10

2

10

1

10 0

10 1

0

20

40

60

80

1

0.2

2

0

20

40

60

80

0.2

0.3

0.4

0.6

0.1

(a) RN18 on CIFAR10

0

20

40

60

80

10

2

10 0

10 2

10 4

10 6

0

20

40

60

80

10

2

10

1

10 0

0

20

40

60

80

10

2

10

1

10 0

(b) LeNet on MNIST

For the CIFAR10 dataset, µ L = 0 . 5014 and µ J = 1 . 7 × 10 -13 . For the MNIST dataset, µ L = 0 . 7192 and µ J = 3 . 7 × 10 -13 . These values are not so large that they are reasonable in practice.

H.2 Extension of I 2 F to the GS Attack

The derivation of I 2 F is considered in the DGL attack as defined in Eq. (1), but I 2 F can also be applied to the GS attack. Note that the minimizer of the DGL attack is one solution to the GS attack. That means the GS attack can be attained by an optimal DGL attack which is our assumption. Therefore, the DGL attack-based theorem is applicable to the GS attack.

Empirically, we evaluate I lb on GS attack in Figs. 2 and 16. It shows that I lb is linearly correlated to the metrics of RMSE, which proves the utility of I 2 F under GS attack.

H.3 Extension of I 2 F with Prior Knowledge

Our theorem of I 2 F can be extended to take into account the prior knowledge. Consider the inversion optimization problem with prior knowledge as min x L ′ I ( x ; g ) = L I ( x ; g ) + I C ( x ) where I C ( x ) constrains x in the prior space C and L I ( x ; g ) = ∥∇ θ L ( x ; θ ) -g ∥ defined in Eq. (1). Then the optimization problem can be rewritten as min x ∈ C L I ( x ; g ) . Thus, as long as the original image x 0 is in the feasible region defined by I C ( x ) , our Assumption 3.1 and theorems are also applicable. Intuitively, a good regularization should satisfy the requirement, otherwise, it will unreasonably reject the correct recovery.

10

2

10

1

10 0

(matrix inversion)

10

2

10

1

(a) DGL (

ϵ

= 0

.

1

)

10

3

10

2

10

1

(matrix inversion)

10

2

10

1

(b) DGL (

ϵ

= 1

)

10

4

10

3

10

2

10

1

(matrix inversion)

10

2

10

1

(c) DGL (

ϵ

= 10

)

10

3

10

2

10

1

lb

(matrix norm)

10

2

10

1

(d) DGL (

I

lb

)

10

3

10

2

10

1

10 0

(matrix inversion)

10

3

10

2

10

1

(e) GS (

ϵ

= 0

.

1

)

10

3

10

2

10

1

(matrix inversion)

10

3

10

2

10

1

(f) GS (

ϵ

= 1

)

10

4

10

3

10

2

10

1

(matrix inversion)

10

3

10

2

10

1

(g) GS (

ϵ

= 10

)

10

3

10

2

10

1

lb

(matrix norm)

10

3

10

2

10

1

(h) GS (

I

lb

)

H.4 Discussion with Prior Works

Closest to our work, Hannun et al. (2021) provided a second-order worst-case metric for analyzing privacy attacks. However, our work provides novel contributions both on technique and implications which essentially root from the proposed I 2 F metric.

Technical difference. First, we focus on a different scope against (Hannun et al., 2021). Hannun et al. (2021) proposed Fisher information loss (FIL) to measure the information leakage risk in the context of model inversion and attribute inference, e.g., only the attribute inference is considered in their experiments. Instead, we evaluate the privacy risk under gradient inversion attacks. Second, our metric is more scalable and applicable to large models. For example, in Eq. (18) in (Hannun et al., 2021), the inverse of the Hessian matrix needs to be calculated even when quantifying the information leakage of only one sample, which is inefficient and intractable for large models. Because of the computation inefficiency, only linear regression and logistic regression models are considered in their theories and experiments. Instead, we verify the feasibility of I 2 F in much larger models like ResNet152 on ImageNet in Fig. 3.

Our new findings. First, though the unfairness of information leakage of different samples was discussed in (Hannun et al., 2021), we investigate the issue in a different attack method and justify the commonness of the unfairness in different attacks. Second, we additionally provide other insights than (Hannun et al., 2021). For instance, the influence on gradient inversion of different initialization methods is studied. We also find the influence of perturbations is not equivalent even in the same noise scale. We believe these insights are also critical to the privacy and security community, especially in the area of gradient inversion.

Besides, (Guo et al., 2022; Hayes et al., 2023) propose to bound the reconstruction attack in terms of the attack success rate and the expectation of the L 2 distance between the recovered and original image. Nevertheless, their conclusions are based on the differential privacy (DP) quantification framework so it is hard to analyze the influence of other defense mechanisms such as gradient pruning and arbitrary noise. Also, they bound the privacy risk from the statistical sense raised by the randomness of DP, while our work can evaluate the sample-wise worst-case privacy risk at any time during the model training. Moreover, they assume the access of the attacker to the remaining samples (except for the privacy sample) or the multi-round gradient, which is not practical in real gradient inversion scenarios.

H.5 Discussions about the Worst-case Assumption

Our work is mainly built upon the Assumption 3.1 that there is only a unique minimizer for L I ( x ; g ) given a gradient vector g . Because of the hardness of optimizing a non-linear objective in attack (Eq. (1)), the worst-case may not be reachable in practice. Here, we discuss when the assumption has to be relaxed and why our method is still applicable. In addition, we emphasize that a stronger attacker exists theoretically, resulting in the necessity of a worst-case assumption.

Non-bijective inversion mapping G r ( g ) . G r ( g ) could be non-bijective when the loss function is non-convex. In other words, given the same g , the output of G r ( g ) could include multiple choices. We want to argue that this case does not conflict with our assumption. Consider an attack given the exact gradient of a sample. Note that the sample itself is a solution to Eq. (1). Thus, given the exact gradient of a sample, the attack can exactly recover the sample. Even if the solution is non-unique, we can still essentially assume the attack can attain the sample in the worst case.

Optimizing the gradient inversion objective may not converge to the original image. Note that the original image is always an optimal solution for the inversion loss in Eq. (1). Even though the convergence is not guaranteed, there always exists an algorithm that can converge to the original image. To our best knowledge, there is no evidence to show the attack cannot approach the worst case where the original input is recovered. Instead, empirical results have shown that the images can be recovered almost perfectly (Geiping et al., 2020). Thus, due to the sensitivity of privacy, a worst-case assumption is necessary to strictly bound possible privacy risks with arbitrary strong attacks, which is commonly imposed by the literature (Dwork, 2006; Abadi et al., 2016).

I Realistic Impact

Federated learning (FL) (McMahan et al., 2017) is a popular distributed training paradigm that benefits from the data and computation sources from multiple clients. As a principle of FL, clients will upload the local gradient based on the private data to the server for the concerns of data privacy and safety, instead of directly sharing the private data of each client. However, recent works (Geiping et al., 2020; Zhu et al., 2019) show that an attacker, who has access to the local gradient (e.g., a malicious server), can leverage the local gradient to recover the private data of the clients, which we call Deep Gradient Leakage (DGL). Even with large models like Transformer, the attacker can still successfully recover the private data given the gradient (Hatamizadeh et al., 2022).

Auditing potential privacy risks is essentially desired for privacy-sensitive applications, including but not limited to finance (Long et al., 2020), healthcare (Antunes et al., 2022; Xu et al., 2021), and clinical data (Dayan et al., 2021; Roth et al., 2020). Such privacy concerns have been discussed extensively in previous work. For instance, Zhang et al. (2023) raises the privacy concern of FL in financial crime detection while (Kaissis et al., 2021; Li et al., 2023) discusses it in the medical and healthcare applications, respectively.

To echo the demands for privacy risk auditing, we provide a fundamental tool for bounding the worst-case risks of DGL. We show multiple insights, for example, the unfairness of privacy protection using random noise defense. Thus, we expect our work can call the attention of the community to the privacy concern raised by DGL, especially the worst-case instance-level privacy risk. Moreover, We expect I 2 F to be a keystone for designing more powerful defense mechanisms.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\n**Key Contribution:** This work proposes a new method for analysis of privacy risk in the deep leakage from gradients attack, which does not rely on assumptions of the model architecture or attack optimization method.\n\n**Approach:** The work defines an “Inverse Influence Function” (I2F) which is able to determine some metric of privacy risk given an applied defense to gradients, irrespective of the model architecture or attack optimization technique. The authors use their framework to identify that the eigenvalues of the matrix $JJ^T$ greatly impacts the MSE of any possible recoveries - in particular, the paper identifies that the smallest eigenvalues can greatly reduce the risks of leakage.\n\n**Evaluation:** The paper evaluates their approach using LeNet and ResNet18, trained on MNIST and CIFAR10. They consider the impacts of different eigenvalues on the predicted MSE in their I2F framework, and find that the most risky samples (i.e. lowest MSE recoveries) tend to be those where the jacobian had large eigenvalues. The paper also evaluates the impacts of different model initialization schemes on recovery MSE.\n\nSTRENGTHS:\n**Originality:**\nThe paper reframes analysis of DLG attacks in terms of prior work by Koh and Liang. In this way, it is an original framing of the DLG problem, using established techniques. \n\n**Quality:**\nThe paper is of good quality, providing interesting conclusions and analysis of the I2F approach.\n\n**Clarity:**\nThe paper is well-written and clear - the mathematical steps are simple and easy to follow, with intuitive results.\n\n**Significance:**\nThe paper builds a simple yet intuitive mathematical framework to estimate privacy risk in DLG attacks. The key significance here is in the simplicity and efficiency of the approach, which allows it to be easily used in practice.\n\nWEAKNESSES:\n**Originality:** The paper claims that prior work by  Fan et al. (2020) is not generalizable to different architectures, such as convolutional networks. However, in Fan et al. (2020) Appendix A it is established that such layers can be re-written as fully-connected ones. It is unclear if the paper’s claims about the limitations of prior approaches are true. A more in-depth comparison with prior would help establish the novelty of the paper.\n\n**Limited Evaluation:** The scope of the evaluation could be broadened to further strengthen the paper’s conclusions. It would be interesting to see how the estimation of MSE changes with respect to training iterations - prior work (as cited in the paper) demonstrates that trained models are more susceptible to the DLG attack, and hence justifying this empirical finding with the I2F framework would lend more credence to its utility. Moreover, the paper only considers two defenses: gradients perturbation, and mixing of training samples. However, the paper could be stronger if it considered more defenses e.g. gradient pruning, which is a commonly considered defense.\n\n**Limited Applicability to Realistic Scenarios:** The paper cites Huang et al. 2021 and mentions that BN statistics are needed for a realistic inversion. However, it is unclear how the lack of BN statistics impacts the I2F framework’s utility in a realistic scenario.\n\nQUESTIONS:\n- (40) What does “converging to the optimal attack” mean in this sentence? Do you mean a perfect recovery of the training data? Or that the optimization method used in the attack itself is optimal?\n\n- (145) Only considering MSE recovery leaves out situations where the recovered image is semantically the same as the private image, but differs in individual pixels. Have you considered alternative metrics which are less sensitive to individual pixels e.g. total variation? \n\n\n**Minor nits:**\nThe original attack is referred to as  “Deep Leakage from Gradients” - is there a reason you chose a different terminology in your paper?\n\nLIMITATIONS:\nThe authors sufficiently addressed the potential social impact and lmiitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

comments: SUMMARY:
This paper proposes inverse influence function (IIF), the indicator of how reconstructed input (from gradient inversion attack) changes with respect to a gradient change. This function can be simply formulated using Jacobian and gradient change. The correlation between the proposed measure and reconstruction quality is demonstrated both theoretically and empirically. Also, IIF can predict the vulnerability of gaussian perturbation (probably for privacy protection) based on Jacobian (or sample). Using this fact, the sample whose Jacobian has larger maximum eigenvalue is considered to be still unsafe from perturbation-based protection. This fact is also empirically demonstrated on MNIST. Various model initialization schemes are also analyzed using eigenvalues of Jacobian.

STRENGTHS:
1. The paper is easy to follow.
2. The formulation of IIF is novel and interesting.
3. The correlation between the proposed measure and reconstruction quality is empirically demonstrated on several datasets
4. Both theoretical and experimental analyses on Gaussian perturbation based privacy protection using IIF is novel and important.

WEAKNESSES:
1. There is no experimental result on large-scale dataset and large DNNs.
2. The worst-case assumption can be very strong in some cases when optimal point from the attack does not reach ground truth.
3. The theory is strongly based on L2 distance between gradients. How about GS attack case, which is based on cosine-similarity loss?

QUESTIONS:
Please see the weaknesses for basic questions.
An another question: what is the conclusion from several analyses on model initialization or perturbation-based protection using IIF?

LIMITATIONS:
Limitation is included in the submission.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose Inversion Influence Function ($I^2F$), a closed-form lower-bound approximation that estimates the recovery $L_2$-norm caused by gradient perturbation in gradient inversion attacks. Detailed mathematical proof and experiments are provided, with comparisons of privacy vulnerability with regard to data, model, perturbation, and attack methods.\n\nSTRENGTHS:\n- The theorem is supported with detailed math proof and experiment results.\n- $I^2F$ gives a good approximation of privacy vulnerability with regard to data, model, perturbation, and attack methods. Most importantly, one can use it to find a theoretically optimal direction to perturb the gradient in federated learning.\n\nWEAKNESSES:\n- In Lemma B.3. (appendix), $\\nabla_x L_I(x;g) = \\nabla_x \\|\\| \\nabla_\\theta L(x, \\theta) - g \\|\\|^2 = 2 \\nabla_x \\nabla_\\theta L(x, \\theta) (\\nabla_\\theta L(x, \\theta) - g)$, the factor $2$ is missing.\n- For figure 7 in the appendix, there's no explanation of what different lines mean, and it makes no sense to compare the running time of power iteration to that of inversion attack.\n- There should be quoted conclusions/studies on the validity of assumptions, especially the actual value of $\\mu_L$ and $\\mu_J$. If they just exist but take very large values, Eq. (7) will be nonsense.\n- $I_{lb}$ in Fig. (1) is an approximation of Eq. (4) and is different from  $I_{lb}$ defined in Eq. (5), and the computation speedup described in \"Efficient Evaluation\" is not used (only the convergence speed is evaluated).\n- $L_2$-norm is used in almost all math equations, but the experiments use RMSE/MSE, which makes it hard to tell how tight the approximated lower bound is.\n- In Sec 5.3, there exist both $\\sigma(\\theta^Tx)-1$ and $\\sigma(\\theta^Tx)-b$, which looks like an incomplete replacement.\n\nQUESTIONS:\n- What is the perturbation scale used in Fig. (4)?\n- In Fig. (5), are gradient perturbations sampled from Gaussian noise, or do the follow the direction of the eigenvector with the smallest eigenvalue?\n\nLIMITATIONS:\nThe proposed $I^2F$ relies on 5 assumptions that may not hold true. Out of these assumptions, 3.2  is satisfied by common loss functions, 3.3 is bypassed by \"Extension to singular Jacobians\". However, other assumptions, as well as the variables they introduce ($\\mu_J$ and $\\mu_L$), have not been well discussed.\n\n> After rebuttal\n\nWe found the proposed method partially overlaps with [the Fisher Information Loss](https://arxiv.org/pdf/2102.11673.pdf), which weakens its technical novelty. However, there are still differences between the two works in terms of the lower bounds and algorithms they arrive at. I agree with AC that this work could be accepted conditioned on that a more thorough discussion of prior works is included in its final version.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes to leverage influence function as a tool for understanding and analyzing the privacy risk in gradient leakage by connecting the private gradients with the recovered images. Inversion Influence Function (I^2F) is introduced as an efficient approximation of deep leakage attacks. Theoretical justification is provided for this approximation and empirical analysis is conducted on two image datasets (MNIST and CIFAR10).\n\nSTRENGTHS:\n1. Understanding privacy leakage through gradients is an important and timely topic.\n2. It is an original work and perhaps the first to leverage influence function to perform analysis on deep gradient leakage.\n3. The analysis provides an explanation for sample- and class-wise variance in reconstruction quality (referred to as “unfair privacy” in the paper).\n\nWEAKNESSES:\n1. For complex neural networks with non-convex loss functions, the inversion mapping Gr may not be bijective, e.g., there might exist multiple x that induce a similar gradient.\n2. The proposed inversion influence function based on first-order Taylor requires the added perturbation to be infinitesimal to be accurate. Although this is partially justified through Theorem 3.1, it can be seen from Fig. 1 that the lower bound is violated at larger noise values.\n3. For singular Jacobians, a constant needs to be added for numerical stability, which has to be tuned for each specific dataset.\n4. From what the reviewer understands, the performed analysis through influence function only considered an adversary who observes the (noisy) gradients and tries to perform inversion by solving the gradient matching optimization problem. In practice, a sophisticated attacker may leverage prior knowledge to improve the reconstruction. The experiments only involved attacks with no/weak prior (e.g., TV), it would be interesting to see how well the proposed metric approximates the reconstruction quality of a strong biased attacker, e.g., using GAN as prior.\n5. In the batch recovery case, the upper bound by decomposing into individual gradient inversion might be too loose.\n6. Not a weakness but some discussion and comparisons between the proposed lower bound and prior work on lower bounding reconstruction MSE would be nice to have, e.g., [R1] using estimation theory and [R2] using DP-SGD.\n[R1] Guo, Chuan, et al. \"Bounding training data reconstruction in private (deep) learning.\" International Conference on Machine Learning. PMLR, 2022.\n[R2] Hayes, Jamie, Saeed Mahloujifar, and Borja Balle. \"Bounding Training Data Reconstruction in DP-SGD.\" arXiv preprint arXiv:2302.07225 (2023).\n\nQUESTIONS:\n1. It can be seen from Fig. 2 that Cifar10 samples induce a larger variance compared to MNIST. Is this an artifact of Cifar10’s data distribution being more complex?\n2. What are the variances of the Gaussian noise used to produce Fig. 1 & 2?\n3. How much data is needed to tune the epsilon constant? How much impact does the constant have on the results? E.g., how would I^2F perform if an epsilon constant tuned on a different dataset is used?\n\nLIMITATIONS:\nThe limitations and social impact have been discussed in the last section.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims to understand how private information leaks from gradients in model training.\nTo this end, the authors propose to use influence analysis to analyze how gradient perturbations can affect the quality of samples reconstructed from private gradients.\nSpecifically, they first prove that under some assumptions, the privacy leaked from a perturbed gradient can be strictly characterized by a novel Inversion Influence Function (I2F), which is a function that takes the gradient perturbation as input.\nWith I2F, the authors then gain insights about DGL, such as (1) common random noise (e.g., Gaussian noise) can not provide uniformly fair privacy protection for each gradient, (2) future protection for DGL can be developed based on the eigenvalues of the Jacobian matrices of the model.\n\nSTRENGTHS:\nThis paper makes the first step to address an important and fundamental problem, i.e., decoupling the black-box behavior of data gradient leakage. The authors make non-trivial contributions in various aspects:\n\n1. The authors design a novel influence function named I2F based on non-trivial influence analysis. It can strictly characterize the privacy protection provided by gradient perturbation, which gives the first theoretical result about how to analyze gradient privacy leakage.\n\n2. Beyond theoretical analysis, the authors also conduct empirical studies to justify the feasibility of using I2F to study DGL in practice.\n\n3. Based on I2F, the authors found that the singular values of model Jacobian with respect to single inputs could be a good indicator for representing privacy protection strengths, which then illustrates that the gradient privacy protection provided by random noise (which is commonly used in practice) could be unfair to different training data.\n\n4. The paper first establishes the connection between model Jacobian and gradient privacy protection, which suggests that future defense mechanisms can be designed starting from the singular values of model Jacobian.\n\nWEAKNESSES:\n1. The main weakness is that the experiments could be not comprehensive enough to justify the effectiveness of I2F. Specifically:\n    - Only two datasets, MNIST and CIFAR-10, are involved in the experiments. Furthermore, one of the datasets MNIST could be too simple for illustrating representative results.\n\n    - Only a few models are used in the experiments and most of them could be too small.\n\n2. Suggestion: I think it is not necessary to use $\\mathcal I_{\\mathrm{lb}}$ as a replacement to approximate I2F since the calculation of I2F seems not really costly. Specifically, the main calculation cost would be calculating the matrix inversion of $JJ^T$. Because $JJ^T$ is (assumed to be) positive definite, you can easily use `numpy.linalg.eigh()` to calculate the matrix inverse, which is usually calculation efficient in practice.\n\n======== After Rebuttal ========\n\nScore has been raised based on the authors response.\n\nQUESTIONS:\nNone.\n\nLIMITATIONS:\nThis paper contributes non-trivially to understanding how privacy leaks from gradients.\nAlthough the empirical analysis could be a little weak, I think this work will significantly contribute to the ML security community and therefore should be accepted.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-15"}