SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \left\| \nabla_ {x} \| \nabla_ {\theta} L (x, \theta) \| ^ {2} - \nabla_ {x} \| \nabla_ {\theta} L (x ^ {\prime}, \theta) \| ^ {2} \right\| \\ = \left\| \nabla_ {x} \nabla_ {\theta} L (x, \theta) \nabla_ {\theta} L (x, \theta) - \nabla_ {x} \nabla_ {\theta} L (x, \theta) \nabla_ {\theta} L \left(x ^ {\prime}, \theta\right) \right. \\ + \nabla_ {x} \nabla_ {\theta} L (x, \theta) \nabla_ {\theta} L \left(x ^ {\prime}, \theta\right) - \nabla_ {x} \nabla_ {\theta} L \left(x ^ {\prime}, \theta\right) \nabla_ {\theta} L \left(x ^ {\prime}, \theta\right) \| \\ \leq \left\| \nabla_ {x} \nabla_ {\theta} L (x, \theta) \right\| \left\| \nabla_ {\theta} L (x, \theta) - \nabla_ {\theta} L \left(x ^ {\prime}, \theta\right) \right\| + \left\| \nabla_ {x} \nabla_ {\theta} L \left(x ^ {\prime}, \theta\right) - \nabla_ {x} \nabla_ {\theta} L (x, \theta) \right\| \left\| \nabla_ {\theta} L \left(x ^ {\prime}, \theta\right) \right\|. \\ \end{array}
$$
Using Assumption 3.4 and 3.5 directly lead to
$$
\left\| \nabla_ {x} \| \nabla_ {\theta} L (x, \theta) \| ^ {2} - \nabla_ {x} \| \nabla_ {\theta} L (x ^ {\prime}, \theta) \| ^ {2} \right\| \leq (\mu_ {L} \| \nabla_ {x} \nabla_ {\theta} L (x, \theta) \| + \mu_ {J} \| \nabla_ {\theta} L (x ^ {\prime}, \theta) \|) \| x - x ^ {\prime} \|.
$$
Lemma B.2.  $\left\| \nabla_{x}(\nabla_{\theta}^{\top}L(x,\theta)g) - \nabla_{x}(\nabla_{\theta}^{\top}L(x',\theta)g)\right\| \leq \mu_{J}\| g\| \| x - x'\|$
Proof. By Assumption 3.4, we have
$$
\begin{array}{l} \left\| \nabla_ {x} (\nabla_ {\theta} ^ {\top} L (x, \theta) g) - \nabla_ {x} (\nabla_ {\theta} ^ {\top} L (x ^ {\prime}, \theta) g) \right\| \\ \leq \left\| \nabla_ {x} \nabla_ {\theta} L (x, \theta) - \nabla_ {x} \nabla_ {\theta} L \left(x ^ {\prime}, \theta\right) \right\| \| g \| \\ \leq \mu_ {J} \| g \| \| x - x ^ {\prime} \|. \\ \end{array}
$$
Lemma B.3. The inversion loss  $L_{I}(x;g)$  defined satisfies  $\| \nabla_x L_I(x;g) - \nabla_x L_I(x';g)\| \leq \mu \| x - x'\|$  where
$$
\mu (x, x ^ {\prime}, \theta , g) = \mu_ {L} \| \nabla_ {x} \nabla_ {\theta} L (x, \theta) \| + \mu_ {J} \| \nabla_ {\theta} L \left(x ^ {\prime}, \theta\right) \| + \mu_ {J} \| g \|. \tag {12}
$$
$$
\begin{array}{l} \nabla_ {x} L _ {I} (x; g) = 2 \nabla_ {x} \nabla_ {\theta} L (x, \theta) (\nabla_ {\theta} L (x, \theta) - g) \\ = 2 \nabla_ {x} \| \nabla_ {\theta} L (x, \theta) \| ^ {2} - 2 \nabla_ {x} \nabla_ {\theta} L (x, \theta) g, \\ \end{array}
$$
we can derive
$$
\begin{array}{l} \left\| \nabla_ {x} L _ {I} (x; g) - \nabla_ {x} L _ {I} \left(x ^ {\prime}; g\right) \right\| \\ = 2 \left\| \nabla_ {x} \| \nabla_ {\theta} L (x, \theta) \| ^ {2} - \nabla_ {x} \| \nabla_ {\theta} L (x ^ {\prime}, \theta) \| ^ {2} - \left[ \nabla_ {x} \nabla_ {\theta} ^ {\top} L (x, \theta) - \nabla_ {x} \nabla_ {\theta} ^ {\top} L (x ^ {\prime}, \theta) \right] g \right\| \\ \leq 2 \left\| \nabla_ {x} \| \nabla_ {\theta} L (x, \theta) \| ^ {2} - \nabla_ {x} \| \nabla_ {\theta} L (x ^ {\prime}, \theta) \| ^ {2} \right\| + 2 \left\| \left[ \nabla_ {x} \nabla_ {\theta} ^ {\top} L (x, \theta) - \nabla_ {x} \nabla_ {\theta} ^ {\top} L (x ^ {\prime}, \theta) \right] g \right\|. \\ \end{array}
$$
By Lemma B.1 and Lemma B.2, we have
$$
\left\| \nabla_ {x} \| \nabla_ {\theta} L (x, \theta) \| ^ {2} - \nabla_ {x} \| \nabla_ {\theta} L \left(x ^ {\prime}, \theta\right) \| ^ {2} \right\| \leq \mu_ {1} \| x - x ^ {\prime} \|,
$$
$$
\left\| \nabla_ {x} \left(\nabla_ {\theta} ^ {\top} L (x, \theta) g\right) - \nabla_ {x} \left(\nabla_ {\theta} ^ {\top} L \left(x ^ {\prime}, \theta\right) g\right) \right\| \leq \mu_ {2} \| x - x ^ {\prime} \|.
$$
where  $\mu_{1} = \mu_{L}\| \nabla_{x}\nabla_{\theta}L(x,\theta)\| +\mu_{J}\| \nabla_{\theta}L(x^{\prime},\theta)\|$  and  $\mu_{2} = \mu_{J}\| g\|$ . Let  $\mu = 2\mu_{1} + 2\mu_{2}$ . Then we can get
$$
\left\| \nabla_ {x} L _ {I} (x; g) - \nabla_ {x} L _ {I} \left(x ^ {\prime}; g\right) \right\| \leq \mu \| x - x ^ {\prime} \|.
$$
Theorem B.1 (Restated from Theorem 3.1). Let  $x_0$  be the private data and  $g_0 \triangleq \nabla_\theta L(x_0, \theta)$  be its corresponding gradient which is treated as a constant. If Assumption 3.4 and 3.5 hold, then the square root of the recovery RMSE satisfies:
$$
\left\| x _ {0} - G _ {r} \left(g _ {0} + \delta\right) \right\| \geq \frac {\| J \delta \|}{\mu_ {L} \| J \| + 2 \mu_ {J} \| g _ {0} + \delta \|}, \tag {13}
$$
Proof. Utilize the stationary condition  $\nabla_x L_I(x_g^*; g) = 0$  and Lemma B.3 to obtain
$$
\left\| \nabla_ {x} L _ {I} (x; g) \right\| \leq \mu \left(x, x _ {g} ^ {*}, \theta , g\right) \left\| x - x _ {g} ^ {*} \right\|, \forall x.
$$
As  $x_0$  is the private sample whose gradient is  $g_0 \triangleq \nabla_\theta L(x_0, \theta)$ , then we have
$$
\left\| x _ {0} - G _ {r} \left(g _ {0} + \delta\right) \right\| \geq \frac {1}{\mu \left(x _ {0} , x _ {g _ {0} + \delta} ^ {*} , \theta , g _ {0} + \delta\right)} \| \nabla_ {x} L _ {I} \left(x _ {0}; g _ {0} + \delta\right) \|
$$
$$
\begin{array}{l} \nabla_ {x} L _ {I} \left(x _ {0}; g _ {0} + \delta\right) = 2 \nabla_ {x} \nabla_ {\theta} L \left(x _ {0}, \theta\right) \left(\nabla_ {\theta} L \left(x _ {0}, \theta\right) - g _ {0} - \delta\right) \\ = 2 \nabla_ {x} \nabla_ {\theta} L (x _ {0}, \theta) \delta , \\ \end{array}
$$
we can attain
$$
\left\| x _ {0} - G _ {r} \left(g _ {0} + \delta\right) \right\| \geq \frac {2}{\mu} \left\| \nabla_ {x} \nabla_ {\theta} L \left(x _ {0}, \theta\right) \delta \right\|.
$$
With  $\nabla_{\theta}L(x_{g_0 + \delta}^*,\theta) = g_0 + \delta$  we can obtain
$$
\begin{array}{l} \mu \left(x _ {0}, x _ {g _ {0} + \delta} ^ {*}, \theta , g _ {0} + \delta\right) = 2 \mu_ {L} \left\| \nabla_ {x} \nabla_ {\theta} L (x _ {0}, \theta) \right\| + 2 \mu_ {J} \left\| \nabla_ {\theta} L \left(x _ {g _ {0} + \delta} ^ {*}, \theta\right) \right\| + 2 \mu_ {J} \| g _ {0} + \delta \| \\ = 2 \mu_ {L} \| J \| + 4 \mu_ {J} \| g _ {0} + \delta \|. \\ \end{array}
$$
Model architectures. The linear model we use is a matrix that maps the input data into a vector. The LeNet model is a convolutional neural network with 4 convolutional layers and 1 fully connected layer. We use the modified version following previous privacy papers Sun et al. (2020), whose detailed structure is in Table 1. ResNet18 is a popular deep convolutional network in computer vision with batch-normalization and residual layers (He et al., 2015a). Cross entropy loss is used as the loss function in all the experiments.
Experimental settings. We conduct two kinds of attacks in our paper: DGL and GS attacks. The learning rate of the two attacks is 0.1 and we use Adam as the optimizer. To consider a more powerful attack, only a single image is reconstructed in each inversion. When inverting LeNet, we uniformly initialize the model parameters in the range of  $[-0.5, 0.5]$  as (Sun et al., 2020) to get a stronger attack. When inverting ResNet18, we use the default initialization method in PyTorch and follow Huang et al. (2021) to use BN statistics as an additional regularization term to conduct a stronger attack.
Table 1: A modified version of LeNet. Conv represents a convolutional layer. FC means a fully-connected layer.
Layers
Conv(in_channels=3, out_channels=12, kernel_size=5)
Conv(in_channels=12, out_channels=12, kernel_size=5)
Conv(in_channels=12, out_channels=12, kernel_size=5)
Conv(in_channels=12, out_channels=12, kernel_size=5)
FC(out_features=10)
(b) ROUGE-L
(c) Google BLEU
(d) Feature MSE
(e) ROUGE-1
(f) ROUGE-L
(g) Google BLEU
Figure 9: Evaluation of  $\mathcal{I}_{lb}$  on BERT (top) and GPT-2 (bottom). A darker color means a larger noise variance. Four metrics are used to evaluate the semantic similarity between the original text and the recovered text.  $\mathcal{I}_{lb}$  is linearly correlated to the four semantic metrics, which means  $\mathcal{I}_{lb}$  can be used to estimate the privacy risk of the private text.
(h) Feature MSE
C.2 Empirical Validation on Language Data
We evaluate the proposed  $\mathrm{I}^2\mathrm{F}$  metric on BERT (Devlin et al., 2018) and GPT-2 (Radford et al., 2019), which are popular language models in natural language processing. We use TAG (Deng et al., 2021), which is an attack on Transformer-based language models based on the  $L_{1}$  and  $L_{2}$  distance between the original gradient and the dummy gradient, and follow the code in https://github.com/JonasGeiping/breaching. We use the default setting in the code and iteratively update the input embedding 12,000 times. We randomly sample 70 sentences from WikiText-103 (Merit et al., 2016) as the private text. We use ROUGE-1, ROUGE-L (Lin, 2004), Google BLEU (Wu et al., 2016) and feature MSE to measure the semantic similarity between the original text and the recovered text. ROUGE-1 measures the overlap of 1-grams in the original and recovered text, while ROUGE-L measures the length of the longest common subsequence between two sentences. While ROUGE metrics calculate the reconstruction recall, the Google BLEU score uses as the output the smaller value of the precision and recall of the original and recovered text and has a broader range of the overlap of  $n$ -grams, where  $n = 1, 2, 3, 4$ . Since the above three metrics are discrete and not consistent with our assumptions, we include a continuous metric, the feature MSE, which measures the distance between the final layer's feature of the original and recovered text.
The results are presented in Fig. 9. A darker color indicates a larger noise variance. For each noise variance, we randomly sample the perturbation from a zero-mean Gaussian distribution 5 times and repeat this for 3 different random seeds. It shows that  $\mathrm{I}^2\mathrm{F}$  is correlated to these four metrics, which means  $\mathrm{I}^2\mathrm{F}$  can be used to estimate the privacy risk of text datasets with large language models. The correlation of BERT and GPT-2 between the four metrics has a similar mode. Though ROUGE-1, ROUGE-L and Google BLEU measure the structural similarity of sentences, which consists of discrete tokens,  $\mathrm{I}^2\mathrm{F}$  presents a clear correlation. For the feature MSE, although  $\mathrm{I}^2\mathrm{F}$  has a less distinguished correlation with feature MSE, it can still be utilized to estimate the privacy risk.
C.3 Efficiency
To show the efficiency of computing the  $\mathrm{I}^2\mathrm{F}$  values, we compare it with the GS attack on randomly-picked CIFAR10 images with ResNet18. As the major complexity comes from evaluating the maximal eigenvalue via the power iteration method, we compare the time required for the power iteration against the inversion loss of GS to converge. We notice that the convergence depends on the initialization of the power iteration and the learning rate for DGL. Thus, we repeat power iteration with
(a) Power iteration
(b) Inversion attack
Figure 10: Evaluation of the efficiency of computing  $\lambda_{\mathrm{max}}(JJ^{\top})$  (our method) by power iteration and inversion attack by minimizing inversion loss  $(L_I)$ . Colors in (a) indicate different seeds. Darker colors in (b) indicate larger learning rates. Our method using power iteration can converge faster than direct inversion attacks.
(a) DGL attack
Figure 11: Comparison of the efficiency of computing  $\mathcal{I}_{lb}$  (our method) by power iteration and inversion attack by minimizing inversion loss  $(L_I)$ . Blue bars indicate the time of computing  $\mathcal{I}_{lb}$  while orange bars indicate the time of minimizing inversion loss by DGL and GS. The time ratio of computing  $\mathcal{I}_{lb}$  versus minimizing inversion loss is present above the orange bars. The x-axis are model-dataset pairs sorted by the model scales: MLP-MNIST, MLP-CIFAR10, LeNet-MNIST, LeNet-CIFAR10, RN18-MNIST, RN18-CIFAR10, RN34-CIFAR10, RN50-CIFAR10, RN101-CIFAR10, RN152-CIFAR10, RN152-ImageNet. For large models and datasets, where minimizing inversion loss needs a huge computation overhead,  $\mathcal{I}_{lb}$  can provide an efficient estimation of the privacy risk.
(b) GS attack
5 different seeds. For the inversion attack, we evaluate multiple learning rates (1, 0.5, 0.1, 0.05, 0.01) to show the fastest convergence. Each experiment is repeated 5 times with different random seeds. As shown in Fig. 10, the power iteration method can converge to the maximal eigenvalue within 50 iterations (5 seconds at most). In comparison, the inversion loss demands 3000 more iterations in 150 seconds to fully converge, which is 20 times larger than the power iteration method. Thus, our method can give an accurate and fast approximation of the recovery MSE without the exhaustive whole inversion process.
In Fig. 11, we compare the computation cost of computing  $\mathcal{I}_{lb}$  with minimizing inversion loss. We show that for almost all the models and datasets we evaluate, the time ratio is larger than 1, which means it is more efficient to compute  $\mathcal{I}_{lb}$  than minimize the inversion loss. It indicates that our method is a more efficient way to estimate the privacy risk for most models and datasets, than the empirical inversion attack. Another key point is that, for large models and datasets, such as models larger than RN18 with CIFAR10 or ImageNet, the time ratio is even much larger than 500. When the time consumption of inversion attacks on these models and datasets is huge (about 16 minutes or even longer), our method significantly reduces the computation overheads for estimating the privacy risks.
(a)  $\lambda = 3.90$
(b)  $\lambda = 0.12$
(c)  $\lambda = 0.03$
(d)  $\lambda = 0.0006$
Figure 12: Same perturbation sizes but different protection effects by different eigenvectors of LeNet. Recovered CIFAR10 images associated with different eigenvectors are present. When perturbing with eigenvectors with smaller eigenvalues, the recovered images are more noisy.
(a) 2.60
(b) 2.71
(c) 2.39
(d) 3.97
(e) 0.15
Figure 13: Same perturbation sizes but different protection effects by different eigenvectors of ResNet18. Recovered CIFAR10 images associated with different eigenvalues are present. When perturbing with eigenvectors with smaller eigenvalues, the recovered images are more noisy and lack some semantic information.
(f) 0.28
(g) 0.32
(h) 0.33
C.4 More Visual Results
More images of unequal perturbations. We present in Fig. 12 the recovered CIFAR10 images when perturbing the gradient with eigenvectors with different eigenvalues. When perturbing with eigenvectors with smaller eigenvalues, the recovered images are more noisy, which is consistent with our former observation.
We also present the unequal perturbations with different eigenvectors of ResNet18 on the CIFAR10 dataset in Fig. 13. Even with the same perturbation scale, the eigenvectors with larger eigenvalues provide stronger protection, where the corresponding recovered images are more noisy and lose some semantic information.
More images of unfair privacy protection. We show more results of unfair privacy protection in Fig. 14. The images of digits 5 and 8 can still be recognized by their outlines, while images of digits 7 and 9 are unrecognizable noise.
We also show the unfair privacy protection of ResNet18 on the CIFAR10 dataset in Fig. 15. In this experiment, we also observe a large variance of recovery MSE among samples, indicating sample-wise unfairness. At the class level, we still can find gradients of a few classes to be easily inverted. For example, class 8 has most MSEs lower than the average value.
D  $\mathbf{I}^2\mathbf{F}$  with Gradient Pruning Defense
We present the relationship between the RMSE and  $\mathcal{I}_{lb}$  in Fig. 16. The y-axis is RMSE and the x-axis is  $\mathcal{I}_{lb}$ . It shows  $\mathcal{I}_{lb}$  can be used to estimate the worst-case privacy risk with gradient pruning defense.
E Comparison of  $\mathbf{I}^2\mathbf{F}$  with More Metrics
MSE is a pixel-wise distance that lacks semantic and structural information. To evaluate the effectiveness of  $\mathrm{I}^2\mathrm{F}$  on more metrics, we consider SSIM and LPIPS (Zhang et al., 2018) to measure the
Figure 14: Original (top) and corresponding recovered (bottom) images of LeNet on the MNIST dataset. The gradients are perturbed with Gaussian noise of variance  $10^{-3}$ . The defense is unfair as images of digit 5 and digit 8 can be recognized by the outline.
Figure 15: The sample-wise and class-wise statistics of the GS MSE on the CIFAR10 dataset of ResNet18. The purple lines indicate the average values. Large variances are observed among samples. The original (first row) and recovered (second row) images for the well- and poorly-protected classes are depicted at the bottom. The defense is unfair as some classes, e.g., class 7 (horse) and class 8 (ship), are more vulnerable to inversion attacks.
(a) RN18-CIFAR10-GS
(b) RN18-CIFAR10-DGL
(c) RN18-MNIST-GS
(d) RN18-MNIST-DGL
(e) LeNet-MNIST-GS
(f) LeNet-MNIST-DGL
(g) LeNet-CIFAR10-GS
(h) LeNet-CIFAR10-DGL
Figure 16: RMSE (y-axis) vs.  $\mathcal{I}_{lb}$  (x-axis) with gradient pruning. A darker color indicates a smaller pruning ratio. It shows that  $\mathcal{I}_{lb}$  is a good estimator of RMSE.
structural similarity and semantic distance between the original and recovered images, respectively. The relationship between SSIM and LPIPS is shown in Fig. 17. Since  $\mathcal{I}_{lb}$  aims to lower bound the privacy risk in terms of RMSE,  $\mathcal{I}_{lb}$  does not have the same scale as SSIM and LPIPS. Nevertheless,  $\mathcal{I}_{lb}$  also has a positive correlation between SSIM and LPIPS, which implies that it is a good estimator for the structural similarity and semantic distance between the original and recovered images.
(c) LeNet-MNIST-GS
(d) LeNet-MNIST-DGL
(e) RN18-CIFAR10-GS
(f) RN18-CIFAR10-DGL
(g) LeNet-MNIST-GS
Figure 17:  $\mathcal{I}_{lb}$  is positively correlated with these two metrics and is a good estimator for the structural similarity and semantic distance between the original and recovered images. Darker color indicates higher variance. Top ((a)-(d)): SSIM (y-axis) vs.  $\mathcal{I}_{lb}$  (x-axis). Bottom ((e)-(h)): LPIPS (y-axis) vs.  $\mathcal{I}_{lb}$  (x-axis). A higher SSIM and a lower LPIPS indicate a higher privacy risk.
(h) LeNet-MNIST-DGL
F Dynamics of  $\mathcal{I}_{lb}$  During Training
Previous existing empirical results show that privacy risk decreases by training epochs (Balunovic et al., 2022; Geiping et al., 2020). We evaluate the dynamics of  $\mathcal{I}_{lb}$ , RMSE with DGL attack and RMSE during training in Fig. 18. It shows that as the training epoch increases, the  $\mathcal{I}_{lb}$  also has an increasing trend. While the RMSE of RN18 on the CIFAR10 dataset has a similar trend as  $\mathcal{I}_{lb}$ , that of LeNet on the MNIST dataset has a significant rise at the epoch 60, which is due to the slower learning speed of LeNet than RN18. Moreover, almost for all the epochs, there is a sample with low  $\mathcal{I}_{lb}$ , which again emphasizes the unfairness in privacy protection.
G The Impact of  $\epsilon$  on Efficient Matrix Inversion
In Fig. 19, we study the impact of  $\epsilon$  on efficient matrix inversion proposed in Section 4. We evaluate the impact on the LeNet with the MNIST dataset. The y-axis is the RMSE.  $\mathcal{I}$  (matrix inversion) is calculated as defined in Eq. (4).  $\mathcal{I}_{lb}$  (matrix norm) is calculated as defined in Eq. (5). It shows with  $\epsilon \in [1,10]$ ,  $\mathcal{I}_{lb}$  is a lower bound of the RMSE. It also shows that  $\mathcal{I}_{lb}$  is an accurate estimator of  $\mathrm{I}^2\mathrm{F}$ . Thus, we can directly use  $\mathcal{I}_{lb}$  in practice to lower bound the privacy risk to avoid fine-tuning the  $\epsilon$ .
H Discussion
H.1 Validity of Assumption 3.4-3.5
We make assumptions about the Lipschitz continuous Jacobian and gradient in Assumption 3.4 and Assumption 3.5, respectively. These two assumptions are not necessary for  $\mathrm{I}^2\mathrm{F}$  but are only used to provide a theoretical validation of  $\mathrm{I}^2\mathrm{F}$  when the noise  $\delta$  is not infinitesimal. To discuss the validity of these two assumptions in practice, we calculate the value of  $\mu_J$  and  $\mu_L$  of LeNet with two datasets.
(a) RN18 on CIFAR10
Figure 18: Privacy risks decrease by training epochs ( $x$ -axis). Different colors indicate different samples. The y-axis from left to right:  $\mathcal{I}_{lb}$ , RMSE w/DGL attack, RMSE w/GS attack whose smaller values indicate lower risks. The blue line indicates the mean value and the shadow is the variance (some outliers are dropped). The noise is sampled from a Gaussian distribution with a mean zero and variance  $10^{-3}$ .
(b) LeNet on MNIST
For the CIFAR10 dataset,  $\mu_{L} = 0.5014$  and  $\mu_{J} = 1.7\times 10^{-13}$ . For the MNIST dataset,  $\mu_{L} = 0.7192$  and  $\mu_{J} = 3.7\times 10^{-13}$ . These values are not so large that they are reasonable in practice.
H.2 Extension of  $\mathbf{I}^2\mathbf{F}$  to the GS Attack
The derivation of  $\mathrm{I}^2\mathrm{F}$  is considered in the DGL attack as defined in Eq. (1), but  $\mathrm{I}^2\mathrm{F}$  can also be applied to the GS attack. Note that the minimizer of the DGL attack is one solution to the GS attack. That means the GS attack can be attained by an optimal DGL attack which is our assumption. Therefore, the DGL attack-based theorem is applicable to the GS attack.
Empirically, we evaluate  $\mathcal{I}_{lb}$  on GS attack in Figs. 2 and 16. It shows that  $\mathcal{I}_{lb}$  is linearly correlated to the metrics of RMSE, which proves the utility of  $\mathrm{I}^2\mathrm{F}$  under GS attack.
H.3 Extension of  $\mathbf{I}^2\mathbf{F}$  with Prior Knowledge
Our theorem of  $\mathrm{I}^2\mathrm{F}$  can be extended to take into account the prior knowledge. Consider the inversion optimization problem with prior knowledge as  $\min_x L_I'(x;g) = L_I(x;g) + I_C(x)$  where  $I_C(x)$  constrains  $x$  in the prior space  $C$  and  $L_{I}(x;g) = \| \nabla_{\theta}L(x;\theta) - g\|$  defined in Eq. (1). Then the optimization problem can be rewritten as  $\min_{x\in C}L_I(x;g)$ . Thus, as long as the original image  $x_0$  is in the feasible region defined by  $I_C(x)$ , our Assumption 3.1 and theorems are also applicable. Intuitively, a good regularization should satisfy the requirement, otherwise, it will unreasonably reject the correct recovery.
(a) DGL  $(\epsilon = 0.1)$
(b) DGL  $(\epsilon = 1)$
(c) DGL  $(\epsilon = 10)$
(d) DGL  $(\mathcal{I}_{lb})$
(e) GS  $(\epsilon = 0.1)$
(f) GS  $(\epsilon = 1)$
(g) GS  $(\epsilon = 10)$
Figure 19: The impact of the value of  $\epsilon$  is evaluated on LeNet with the MNIST dataset. The y-axis is the RMSE. (a)(b)(c)(d): DGL attack. (e)(f)(g)(h): GS attack. It is observed that  $\mathcal{I}$  (matrix inversion) is effective with  $\epsilon \in [1,10]$  but not  $\epsilon = 0.1$ . It shows that (1) there exists a range of  $\epsilon$  where  $\mathcal{I}_{lb}$  can lower bound the RMSE; (2)  $\mathcal{I}_{lb}$  is an accurate estimator of  $\mathrm{I}^2\mathrm{F}$ , thus we can avoid fine-tuning  $\epsilon$ .
(h) GS  $(\mathcal{I}_{lb})$
H.4 Discussion with Prior Works
Closest to our work, Hannun et al. (2021) provided a second-order worst-case metric for analyzing privacy attacks. However, our work provides novel contributions both on technique and implications which essentially root from the proposed  $\mathrm{I}^2\mathrm{F}$  metric.