SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Haobo Zhang*
Michigan State University  
zhan2060@msu.edu
Junyuan Hong*
Michigan State University  
University of Texas at Austin
jyhong@utexas.edu
Pennsylvania State University mahdavi@cse.psu.edu
Jiayu Zhou
Michigan State University  
jiayuz@msu.edu
Deep Gradient Leakage (DGL) is a highly effective attack that recovers private training images from gradient vectors. This attack casts significant privacy challenges on distributed learning from clients with sensitive data, where clients are required to share gradients. Defending against such attacks requires but lacks an understanding of when and how privacy leakage happens, mostly because of the black-box nature of deep networks. In this paper, we propose a novel Inversion Influence Function  $(\mathrm{I}^2\mathrm{F})$  that establishes a closed-form connection between the recovered images and the private gradients by implicitly solving the DGL problem. Compared to directly solving DGL,  $\mathrm{I}^2\mathrm{F}$  is scalable for analyzing deep networks, requiring only oracle access to gradients and Jacobian-vector products. We empirically demonstrate that  $\mathrm{I}^2\mathrm{F}$  effectively approximated the DGL generally on different model architectures, datasets, modalities, attack implementations, and perturbation-based defenses. With this novel tool, we provide insights into effective gradient perturbation directions, the unfairness of privacy protection, and privacy-preferred model initialization. Our codes are provided in https://github.com/illidanlab/inversion-influence-function.
With the growing demands for more data and distributed computation resources, distributed learning gains its popularity in training large models from massive and distributed data sources (McMahan et al., 2017; Gemulla et al., 2011). While most distributed learning requires participating clients to share only gradients or equivalents and provides data confidentiality, adversaries can exploit such information to infer training data. Deep Gradient Leakage (DGL) showed such a real-world risk in distributed learning of deep networks (Zhu et al., 2019; Jin et al., 2021). Only given a private gradient and the corresponding model parameters, DGL reverse-engineers the gradient and recovers the private images with high quality. The astonishing success of the DGL started an intensive race between privacy defenders (Sun et al., 2020; Wei et al., 2021; Gao et al., 2021; Scheliga et al., 2022) and attackers (Geiping et al., 2020; Zhao et al., 2020; Jin et al., 2021; Jeon et al., 2021; Zhu and Blaschko, 2021; Fowl et al., 2022).
A crucial question to be answered behind the adversarial game is when and how DGL happens, understanding of which is fundamental for designing attacks and defenses. Recent advances in
theoretical analysis shed light on the question: Balunović et al. (2022) unified different attacks into a Bayesian framework, leading to Bayesian optimal adversaries, showing that finding the optimal solution for the DGL problem might be non-trivial and heavily relies on prior knowledge. Pan et al. (2022); Wang et al. (2023b) advanced the understanding of the leakage in a specific class of network architectures, for example, full-connected networks. However, it remains unclear how the privacy leakage happens for a broader range of models, for example, the deep convolutional networks, and there is a pressing need for a model-agnostic analysis.
In this paper, we answer this question by tracing the recovered sample by DGL algorithms back to the private gradient, where the privacy leakage ultimately derives from. To formalize the optimal DGL adversary without practical difficulties in optimization, we ask the counterpart question: what would happen if we slightly change the private gradient?
Answering this question by perturbing the gradient and re-evaluating the attack can be prohibitive due to the difficulty of converging to the optimal attack in a highly non-convex space. Accordingly, we propose a novel Inversion Influence Function  $(\mathrm{I}^2\mathrm{F})$  that provides an analytical description of the connection between gradient inversion and perturbation. Compared to directly solving DGL, the analytical solution in  $\mathrm{I}^2\mathrm{F}$  efficiently scales up for deep networks and requires only oracle access to gradients and Jacobian-vector products. We note that  $\mathrm{I}^2\mathrm{F}$  shares the same spirit of the influence function (Koh and Liang, 2017; Hampel, 1974) that describes how optimal model parameters would change upon the perturbation of a sample and  $\mathrm{I}^2\mathrm{F}$  can be considered as an extension of the influence function from model training to gradient inversion.
The proposed  $\mathrm{I}^2\mathrm{F}$  characterizes leakage through the lens of private gradients and provides a powerful tool and new perspectives to inspect the privacy leakage: (1) First, we find that gradient perturbation is not homogeneous in protecting the privacy and is more effective if it aligns with the Jacobian singular vector with smaller singular values. (2) As the Jacobian hinges on the samples, the variety of their Jacobian structures may result in unfair privacy protection under homogeneous Gaussian noise. (3) We also examine how the initialization of model parameters reshapes the Jocabian and therefore leads to quite distinct privacy risks.
These new insights provide useful tips on how to defend DGL, such as perturbing gradient in specific directions rather than homogeneously, watching the unfairness of protection, and carefully initializing models. We envision such insights could lead to the development of fine-grained privacy protection mechanisms. Overall, our contributions can be summarized as follows. (1) For the first time, we introduce the influence function for analyzing DGL; (2) We show both theoretical and empirical evidence of the effectiveness of the proposed  $\mathrm{I}^2\mathrm{F}$  which efficiently approximates the DGL in different settings; (3) The tool brings in multiple new insights into when and how privacy leakage happens and provides a tool to gauge improved design of attack and defense in the future.
Attack. Deep Gradient Leakage (DGL) is the first practical privacy attack on deep networks (Zhu et al., 2019) by only matching gradients of private and synthetic samples. As accurately optimizing the objective could be non-trivial for nonlinear networks, a line of work has been developed to strengthen the attack (Zhao et al., 2020). Geiping et al. (2020) introduce scaling invariance via cosine similarity. Balunović et al. (2022) summarizes these attacks as a variety of prior choices. The theorem is echoed by empirical studies that show better attacks using advanced image priors (Jeon et al., 2021). It was shown that allowing architecture modification could weigh in higher risks in specific cases (Zhu and Blaschko, 2021; Fowl et al., 2022). Besides the study of gradient leakage in initialized models, recent evidence showed that the well-trained deep network can also leak the private samples (Haim et al., 2022). Yet, such an attack still requires strict conditions to succeed, e.g., small dataset and gradient flow assumptions. Though these attacks were shown to be promising, exactly solving the attack problem for all types of models or data is still time-consuming and can be intractable in reasonable time (Huang et al., 2021).
Defense. Most defense mechanisms introduce a perturbation in the pipeline of gradient computation. 1) Gradient Noise. Since the DGL is centered on reverse engineering gradients, the straightforward defense is to modify the gradient itself. Zhu et al. (2019) investigated a simple defense by Gaussian noising gradients or pruning low-magnitude parameters-based defense. The Gaussian noise was motivated by Differential Privacy Gradient Descent (Dwork, 2006; Abadi et al., 2016), which
guarantees privacy in an aggregated gradient. Except for random perturbation, the perturbation could also be guided by representation sensitivity (Sun et al., 2020) or information bottleneck Scheliga et al. (2022). 2) Sample Noise. The intuition of sample noise is to change the source of the private gradient. For example, Gao et al. (2021) proposed to seek transformations such that the transformed images are hard to be recovered. Without extensive searching, simply using strong random data augmentation, e.g., MixUp (Zhang et al., 2017), can also help defense (Huang et al., 2021). If public data are available, hiding private information inside public samples is also an alternative solution (Huang et al., 2020). Though the defenses showed considerate protection against attacks in many empirical studies, the security still conceptually relied on the expected risks of a non-trivial number of evaluations. Explanation of how well a single-shot protection, for example, noising the gradient, is essential yet remains unclear.
Understanding DGL. There are many recent efforts in understanding when noise-based mechanisms help privacy protection. Though Differential Privacy (DP) provides a theoretical guarantee for general privacy, it describes the worst-case protection and does not utilize the behavior of DGL, motivating a line of studies to understand when and how the DGL happens. Sun et al. (2020) connected the privacy leakage to the representation, which aligns with later work on the importance of representation statistics in batch-normalization layers (Hatamizadeh et al., 2021; Huang et al., 2021). Balunovic et al. (2022) unified different attacks in a Bayesian framework where existing attacks differ mainly on the prior probability. Chen and Campbell (2021) investigated the model structure to identify the key reasons for gradient leakage. Hatamizadeh et al. (2021); Huang et al. (2021) evaluated the DGL in practical settings and showed that BN statistics is critical for inversion. There are efforts advancing theoretical understandings of DGL. Pan et al. (2022) studied the security boundary of ReLU activation functions. In a specific class of fully-connected networks, Wang et al. (2023b) showed that a single gradient query at randomized model parameters like DP protection can be used to reconstruct the private sample. Hayes et al. (2023); Fan et al. (2020) proposed an upper bound of the success probability of the data reconstruction attack and the relative recovery error, respectively, which is considered as the least privacy risk. Instead, we consider the worst-case privacy risk.
Though many useful insights are provided, existing approaches typically focus on specific model architectures. This work provides a new tool for analyzing the DGL without assumptions about the model architectures or attacking optimization. Independent of the model architectures, our main observation is that the Jacobian, the joint derivative of loss w.r.t. input and parameters, is the crux.
3 Inversion Influence Function
In this section, we propose a new tool to unravel the black box of DGL. We consider a general loss function  $L(x,\theta)$  defined on the joint space of data  $\mathcal{X}$  and parameter  $\Theta$ . For simplicity, we assume  $x$  is an image with oracle supervision if considering supervised learning. Let  $x_0$  be the private data and  $g_0\triangleq \nabla_\theta L(x_0,\theta)$  be its corresponding gradient which is treated as a constant. An inversion attack aims at reverse engineering  $g_{0}$  to obtain  $x_0$ . Deep Gradient Leakage (DGL) attacks by directly searching for a synthetic sample  $x$  to reproduce the private gradient  $g$  by  $\nabla_{\theta}L(x,\theta)$  (Zhu et al., 2019). Formally, the DGL is defined as a mapping from a gradient  $g$  to a synthetic sample  $x_{g}^{*}$ :
$$
x _ {g} ^ {*} = G _ {r} (g) \triangleq \underset {x \in \mathcal {X}} {\arg \min } \left\{L _ {I} (x; g) \triangleq \| \nabla_ {\theta} L (x, \theta) - g \| ^ {2} \right\}, \tag {1}
$$
where  $L_{I}$  is the inversion loss matching two gradients,  $\| \cdot \|$  denotes the  $\ell_2$ -norm either for a vector or a matrix. The  $L_{2}$ -norm of a matrix  $A$  is defined by the induced norm  $\| A\| = \sup_{x\neq 0}(\| Ax\| /\| x\|)$ . The Euclidean distance induced by  $\ell_2$ -norm can be replaced by other distances, e.g., cosine similarity (Geiping et al., 2020), which is out of our scope, though.
Our goal is to understand how the gradient of a deep network at input  $x_0$  encloses the information of  $x_0$ . For a linear model, e.g.,  $L(x,\theta) = x^{\top}\theta$ , the gradient is  $x$  and directly exposes the direction of the private sample. However, the task gets complicated when a deep neural network is used and  $L(x,\theta)$  is a nonlinear loss function. The complicated functional makes exactly solving the minimization in Eq. (1) highly non-trivial and requires intensive engineering in hyperparameters and computation (Huang et al., 2021). Therefore, analyses tied to a specific DGL algorithm may not be generalizable. We introduce an assumption of a perfect attacker to achieve a generalizable analysis:
Assumption 3.1. Given a gradient vector  $g$ , there is only a unique minimizer for  $L_{I}(x;g)$ .
Since  $\nabla_{\theta}L(x,\theta)$  is a minimizer for  $G_{r}(\nabla_{\theta}L(x,\theta))$ , Assumption 3.1 implies that  $G_{r}(\nabla_{\theta}L(x,\theta))$  is a perfect inversion attack that recovers  $x$  exactly. The assumption of a perfect attacker considers the worst-case. It allows us to develop a concrete analysis, even though such an attack may not always be feasible in practice, for example, in non-linear deep networks. To start, we outline the desired properties for our privacy analysis. (1) Efficiency: Privacy evaluation should be efficient in terms of computation and memory; (2) Proximity: The alternative should provide a good approximation or a lower bound of the risk, at least in the high-risk region; (3) Generality: The evaluation should be general for different models, datasets, and attacks.
3.1 Perturbing the Private Gradient
To figure out the association between the leakage and the gradient  $g$ , we formalize a counterfactual: what kind of defense can diminish the leakage? A general noise-based defense can be written as  $g = \nabla_{\theta}L(x_0,\theta) + \delta$  where  $\delta$  is a small perturbation. According to Assumption 3.1, a zero  $\delta$  is not private. Instead, we are interested in non-zero  $\delta$  and the recovery error (RE) of the recovered image:
$$
\operatorname {R E} \left(x _ {0}, G _ {r} (g _ {0} + \delta)\right) \triangleq \| x _ {0} - G _ {r} (g _ {0} + \delta) \| \tag {2}
$$
For further analysis, we make two common and essential assumptions as follows.
Assumption 3.2.  $L(x,\theta)$  is twice-differentiable w.r.t.  $x$  and  $\theta$ , i.e.,  $J \triangleq \nabla_x \nabla_\theta L(x,\theta) \in \mathbb{R}^{d_x \times d_\theta}$  exist, where  $d_x$  and  $d_\theta$  are the dimensions of  $x$  and  $\theta$ , respectively.
Assumption 3.3.  $JJ^{\top}$  is invertible.
We then approximate  $G_{r}(g_{0} + \delta)$  by the first-order Taylor expansion:
$$
G _ {r} \left(g _ {0} + \delta\right) \approx G _ {r} \left(g _ {0}\right) + \frac {\partial G _ {r} \left(g _ {0}\right)}{\partial g _ {0}} \delta = x _ {0} + \frac {\partial G _ {r} \left(g _ {0}\right)}{\partial g _ {0}} \delta . \tag {3}
$$
By the implicit function theorem, we can show that  $\frac{\partial G_r(g_0)}{\partial g_0} = (JJ^\top)^{-1}J$  (proof in Appendix B.1). Thus, for a small perturbation  $\delta$ , we can approximate the privacy leakage through DGL by
$$
\text {I n v e s i o n} \left(\mathrm {I} ^ {2} \mathrm {F}\right): \| G _ {r} \left(g _ {0} + \delta\right) - x _ {0} \| \approx \mathcal {I} (\delta ; x _ {0}) \triangleq \left\| \left(J J ^ {\top}\right) ^ {- 1} J \delta \right\|. \tag {4}
$$
The  $\mathrm{I}^2\mathrm{F}$  includes a matrix inversion, computing which may be expensive and unstable for singular matrixes. Thus, we use a tractable lower bound of  $\mathrm{I}^2\mathrm{F}$  as:
$$
\left\| \left(J J ^ {\top}\right) ^ {- 1} J \delta \right\| \geq \frac {\| J \delta \|}{\lambda_ {\max } \left(J J ^ {\top}\right)} \triangleq \mathcal {I} _ {\mathrm {l b}} (\delta ; x _ {0}), \tag {5}
$$
where  $\lambda_{\mathrm{max}}(A)$  denotes the maximal eigenvalues of a matrix  $A$ . Computing the maximal eigenvalue is usually much cheaper than the matrix inversion. We note that the approximation uses Assumption 3.1 and provides a general risk measurement of different attacks.
The lower bound describes how risky the gradient is in the worst case. This follows the similar intuition of Differential Privacy (DP) (Dwork, 2006) to bound the max chance (worst case) of identifying a private sample from statistic aggregation. The proposed lower bound differentiates the DP worst case in that it leverages the problem structure of gradient inversion and therefore characterizes the DGL risk more precisely.
Efficient Evaluation. The evaluation of  $\mathcal{I}$  or its lower bound  $\mathcal{I}_{\mathrm{lb}}$  is non-trivial due to three parts: computation of the Jacobian, the matrix inversion, and eigenvalue. The former two computations can be efficiently evaluated using established techniques, for example, (Koh and Liang, 2017). 1) Efficient evaluation of  $J\delta$ . Instead of computing the dense Jacobian, an efficient alternative is to leverage the Jacobian-vector product, which is similar to the Hessian-vector product. We can efficiently evaluate the product of Jacobian and  $\delta$  by rewriting the product as  $J\delta = \nabla_{x}(\nabla_{\theta}^{\top}L(x_{0},\theta)\delta)$ . Since  $\nabla_{\theta}^{\top}L(x_0,\theta)\delta$  is a scalar, the second derivative w.r.t.  $x$  can be efficiently evaluated by autograd tools, e.g., PyTorch. The computation complexity is equivalent to two times of gradient evaluation in addition to one vector production. 2) Efficient matrix inversion. We can compute  $b\triangleq (JJ^{\top})^{-1}J\delta$  by solving  $\min_b\frac{1}{2}\left\| \delta -J^\top b\right\|^2$ , whose gradient computation only contains two Jacobian-vector products. There are other alternative techniques, for example, Neumann series or stochastic approximation, as discussed in (Koh and Liang, 2017). For brevity, we discuss these alternatives in Appendix A.
In this paper, we use the least square trick and solve the minimization by gradient descent directly. 3) Efficient evaluation of privacy with the norm of Jacobian. Because that  $\| J\| = \sqrt{\lambda_{\max}(JJ^{\top})}$ , we need to compute the maximum eigenvalue of  $JJ^{\top}$ , which can be done efficiently by power iteration and the trick of Jacobian-vector products.
Complexity. Suppose the computation complexity for evaluating gradient is a constant in the order  $\mathcal{O}(d)$  where  $d$  is the maximum between the dimension of  $\theta$  and  $x$ . Then evaluating the Jacobian-vector product is of complexity  $\mathcal{O}(d)$ . By the least square trick with  $T$  iterations, we can compute  $\mathcal{T}$  in  $\mathcal{O}(dT)$  time. The lower bound  $\mathcal{I}_{\mathrm{lb}}$  replaces the computation of matrix inversion with computation on the max eigenvalue. By  $T'$ -step power iteration, the complexity is  $\mathcal{O}(d(T + T'))$ .
Case study on Gaussian perturbation. Gaussian noise is induced from DP-SGD (Abadi et al., 2016) where gradients are noised to avoid privacy leakage. Assuming the perturbation follows a standard Gaussian distribution, we can compute the expectation as follows:
$$
\mathbb {E} \left[ \mathrm {R E} ^ {2} \left(x _ {0}, G _ {r} \left(g _ {0} + \delta\right)\right) \right] \approx \mathbb {E} \left[ \mathcal {I} ^ {2} (\delta ; x _ {0}) \right] = \sum_ {i = 1} ^ {d} \lambda_ {i} ^ {- 1} \left(J ^ {\top} J\right), \tag {6}
$$
where  $\lambda_{i}(J^{\top}J)$  is the  $i$ -th eigenvalue of  $J^{\top}J$ . The above illustrates the critical role of the Jacobian eigenvalues. For a perturbation to offer a good defense and yield a large recovery error, the Jacobian should have at least one small absolute eigenvalue. In other words, the protection is great when the Jacobian is nearly singular or rank deficient. The intuition for the insight is that a large eigenvalue indicates the high sensitivity of gradients on the sample, and therefore an inaccurate synthesized sample will be quickly gauged by the increased inversion loss.
3.2 Theoretic Validation
The proposed  $\mathrm{I}^2\mathrm{F}$  assumes that the perturbation is small enough for an accurate approximation, and yet it is interesting to know how good the approximation is for larger  $\delta$ . To study the scenario, we provide a lower bound of the inversion error on a non-infinitesimal  $\delta$ , using weak Lipschitz assumptions for any two samples  $x$  and  $x'$ .
Assumption 3.4. There exists  $\mu_J > 0$  such that  $\| \nabla_x\nabla_\theta L(x,\theta) - \nabla_x\nabla_\theta L(x',\theta)\| \leq \mu_J\| x - x'\|$ .
Assumption 3.5. There exists  $\mu_L > 0$  such that  $\| \nabla_\theta L(x,\theta) - \nabla_\theta L(x',\theta)\| \leq \mu_L\| x - x'\|$ .
Theorem 3.1. If Assumption 3.4 and 3.5 hold, then the recovery error satisfies:
$$
\left\| x _ {0} - G _ {r} \left(g _ {0} + \delta\right) \right\| \geq \frac {\| J \delta \|}{\mu_ {L} \| J \| + 2 \mu_ {J} \| g _ {0} + \delta \|}, \tag {7}
$$
where  $J = \nabla_{x}\nabla_{\theta}L(x_{0},\theta)$
The proof is available in Appendix B.2. Similar to  $\mathrm{I}^2\mathrm{F}$ , the inversion error could be lower bound using the Jacobian matrix. Eq. (7) matches our lower bound  $\mathcal{I}_{\mathrm{lb}}$  in Eq. (5) with an extra term of gradients. The bound implies that our approximation could be effective as a lower bound even for larger  $\delta$ . Besides, the lower bound provides additional information when  $\delta$  is large: when  $\| \delta \|$  gets infinitely large, the bound converges to  $\Omega (\| J\delta \| /( \mu_J\| \delta \|))$ , which does not scale up to infinity. The intuition is that the  $x$  is not necessarily unbounded for matching an unbounded 'gradient'.
4 Empirical Validation and Extensions
We empirically show that our metric is general for different attacks with different models on different datasets from different modalities.
Setup. We evaluate our metric on two image-classification datasets: MNIST (LeCun, 1998) and CIFAR10 (Krizhevsky et al., 2009). We use a linear model and a non-convex deep model ResNet18 (He et al., 2015a) (RN18) trained with cross-entropy loss. We evaluate our metric on two popular attacks: the DGL attack (Zhu et al., 2019) and the GS attack (Geiping et al., 2020). DGL attack minimizes the  $L_{2}$  distance between the synthesized and ground-truth gradient, while the GS attack maximizes the cosine similarity between the synthesized and ground-truth gradient. We use the Adam optimizer with a learning rate of 0.1 to optimize the dummy data. Per attacking iteration, the dummy data is projected into the range of the ground-truth data [0, 1] with the GS attack. When the DGL attack
(a) Compare different approximation
(b) Compare  $\mathrm{I}^2\mathrm{F}$  to RMSE
Figure 1: Value comparisons attacking ResNet18 on MNIST by DGL, where the grey line indicates the equal values and darker dots imply smaller Gaussian perturbation  $\delta$ . In (a), the y-axis is calculated as defined in Eq. (4) and  $\mathcal{I}_{lb}$  is calculated as defined in Eq. (5).  $\mathrm{I}^2\mathrm{F}$  lower bound  $(\mathcal{I}_{\mathrm{lb}})$  provides a good approximation to the exact value with matrix inversion and to the root of mean square error (RMSE) of recovered images. Instead, removing the denominator in  $\mathcal{I}$  results in overestimated risks.
(a) CIFAR10, DGL, RN18
(b) CIFAR10, GS, RN18
(c) MNIST, GS, RN18
Figure 2:  $\mathrm{I}^2\mathrm{F}$  works under different settings: datasets, attacks, and models. The grey line indicates the equal values, and darker dots imply smaller Gaussian perturbation  $\delta$ .
(d) MNIST, GS, Linear
cannot effectively recover the data, e.g., on noised linear gradients, we only conduct the GS attack to show the highest risk. We use the root of MSE (RMSE) to measure the difference between the ground-truth and recovered images. All the experiments are conducted on one NVIDIA RTX A5000 GPU with the PyTorch framework. More details and extended results are attached in Appendix C.