SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Extension to singular Jacobians. We consider the  $JJ^{\top}$  be singular, which often happens with deep neural networks where many eigenvalues are much smaller than the maximum. In this case, the inversion of  $JJ^{\top}$  does not exist or could result in unbounded outputs. To avoid the numerical instability, we use  $\left\| (JJ^{\top} + \epsilon I)^{-1}J\delta \right\|$  with a constant  $\epsilon$ . In our experiments, the  $\epsilon$  is treated as a hyperparameter and will be tuned on samples to best fit the linear relation between RMSE and  $\mathcal{I}$ .
Batch data. In practice, an attacker may only be able to carry out batch gradient inverting,  $\left\| \frac{1}{n}\sum_{i}^{n}\nabla_{\theta}L(x_{i},\theta) - \frac{1}{n}\sum_{i}^{n}g_{i}\right\|$ , which can be upper bounded by  $\frac{1}{n}\sum_{i}^{n}\|\nabla_{\theta}L(x_{i},\theta) - g_{i}\|$ . Such decomposition can be implemented following (Wen et al., 2022), which modifies the parameters of the final layer to reduce the averaged gradient to an update of a single sample. Thus, we only consider the more severe privacy leakage by individual samples instead of a batch of data.
How well are the influence approximations? In Fig. 1(a), we compare the two approximations of  $\mathrm{I}^2\mathrm{F}$ .  $\mathcal{I}_{\mathrm{lb}}$  relaxes the matrix inversion in  $\mathcal{I}$  to the reciprocal of  $\|JJ^\top\|$ .  $\mathcal{I}_{\mathrm{nom}}$  further drops the denominator and only keeps the nominator  $\|J\delta\|$ . The two figures show that  $\mathcal{I}_{\mathrm{lb}}$  serves as a tight approximation to the real influence while  $\mathcal{I}_{\mathrm{nom}}$  is much smaller than the desired value. In Fig. 1(b), we see that the relaxation  $\mathcal{I}_{\mathrm{lb}}$  is also a good approximation or lower bound for the RMSE.
Validation on different models, datasets, and attacks. In Fig. 2, we show that  $\mathrm{I}^2\mathrm{F}$  can work approximately well in different cases though the approximation accuracy varies by different cases. (1) Attack: In Figs. 2(a) and 2(b), we compare DGL to the Gradient Similarly (GS) (Geiping et al., 2020) that matches gradient using cosine similarity instead of the  $L_{2}$  distance. We see that different attacks do not differ significantly. (2) Dataset: As shown in Figs. 2(b) and 2(c), different datasets could significantly differ regarding attack effectiveness. The relatively high RMSE implies that CIFAR10 tends to be harder to attack due to more complicated features than MNIST. Our metric suggests that there may exist a stronger inversion attack causing even lower RMSE. (3) Model:
${}^{2}\mathrm{\;F} = 1 \times  {10}^{-2}$  . PIPS  $= {0.48}$
$\mathrm{I}^2\mathrm{F} = 5\times 10^{-6}$  LPIPS=0.24
Figure 3: Evaluation of  $\mathrm{I}^2\mathrm{F}$  on ResNet152 and ImageNet. (a): Darker color means larger noise variance. LPIPS is used to evaluate the semantic information of the recovered and original images.  $I_{lb}$  is a good estimator of the semantic distance between the recovered images and original images. (b): Original (top) and recovered (bottom) images with their corresponding  $\mathrm{I}^2\mathrm{F}$  and LPIPS. Images with a lower  $\mathrm{I}^2\mathrm{F}$  also have a smaller LPIPS, which implies a better reconstruction.
$\mathrm{I}^2\mathrm{F} = 4\times 10^{-3}$ $\mathrm{I}^2\mathrm{F} = 1\times 10^{-2}$  LPIPS=0.42 LPIPS=0.48
(b)
$\mathsf{I}^2\mathsf{F} = 1.1$  PIPS  $= 0.66$
Comparing Figs. 2(c) and 2(d), the linear model presents a better correlation than the ResNet18 (RN18). Because the linear model is more likely to be convex, the first-order Taylor expansion in Eq. (3) can approximate its RMSE than one of the deep networks.
Results on Large Models and Datasets. We also evaluate our  $\mathrm{I}^2\mathrm{F}$  metric on ResNet152 with ImageNet. For larger models, the RMSE is no longer a good metric for the recovery evaluation. Even if state-of-the-art attacks are used and the recovered image is visually similar to the original image in Fig. 3(a), the two images are measured to be different by RMSE, due to the visual shift: The dog head is shifted toward the left side. To capture such shifted similarity, we use LPIPS (Zhang et al., 2018) instead, which measures the semantic distance between two images instead of the pixel-to-pixel distance like RMSE. Fig. 3(a) shows that I2F is correlated to LPIPS using large models and image scales. This implies that I2F is a good estimator of recovery similarity. In Fig. 3(b), original images with a lower I2F also have a smaller LPIPS, which means a better reconstruction. Recovered images on the right (the German shepherd and the panda) cannot be recognized while those on the left (the owl and the beagle) still have enough semantic information for humans to recognize.
Results on Language Models and Datasets. Since the input space of language models is not continuous like images, current attacks on images cannot be directly transferred to text. Besides, continuously optimized tokens or embeddings should be projected back to the discrete text, which induces a gap between the optimal and the original text. Thus, the investigation of privacy risks in text is an interesting yet non-trivial problem. We evaluate the proposed  $\mathrm{I}^2\mathrm{F}$  metric on BERT (Devlin et al., 2018) and GPT-2 (Radford et al., 2019) with TAG attack (Deng et al., 2021), which minimizes  $L_{2}$  and  $L_{1}$  distance between gradients of the original and recovered images. We use ROUGE-L (Lin, 2004) and Google BLEU (Wu et al., 2016) to measure the semantic similarity between the original text and the recovered text. More experimental details and results with ROUGE-1 and feature MSE are provided in C.2. The results are presented in Fig. 4. A darker color indicates a larger noise variance. Since the ROUGE-L and Google BLEU measure the semantic similarity of the text pair while our metric estimates the difference, two semantic metrics are negatively correlated to  $\mathcal{I}_{lb}$ . The results demonstrate the effectiveness of our metric in evaluating the privacy risks of discrete data.
5 When Does Privacy Leakage Happen?
Guided by the formulation of  $\mathrm{I}^2\mathrm{F}$ , we provide insights on when privacy leakage happens based on the proposed metric. Our insights are rooted in the properties of Jacobian. Since we want to consider the whole property of the Jacobian matrix, we choose a shallow convolutional neural network LeNet (LeCun et al., 1998) to trade off the utility and efficiency of experiments by default.
5.1 Perturbations Directions Are Not Equivalent
**Implication on choosing  $\delta$ .** Eq. (4) clearly implies that the perturbation is not equal in different directions. Decomposing  $J = U\Sigma V^{\top}$  using Singular Value Decomposition (SVD), we obtain  $\mathcal{I}(\delta; x_0) = \left\|U\Sigma^{-1}V^{\top}\delta\right\|$ . Thus,  $\delta$  tends to yield a larger  $\mathrm{I}^2\mathrm{F}$  value if it aligns with the directions of small eigenvalues of  $JJ^{\top}$ .
(a) ROUGE-L
(b) Google BLEU
(d) Google BLEU
(b) MNIST
Figure 4: Evaluation of  $\mathcal{I}_{lb}$  on BERT (a-b) and GPT-2 (c-d). A darker color means a larger noise variance. ROUGE-L and Google BLEU are used to evaluate the semantic similarity between the original text and the recovered text.  $\mathcal{I}_{lb}$  is linearly correlated to the two semantic metrics, which means  $\mathcal{I}_{lb}$  can be used to estimate the privacy risk of the private text.
Singular values of Jacobian of deep networks. In Fig. 5, we examine the eigenvalues of LeNet on the MNIST and CIFAR10 datasets. On both datasets, there are always a few eigenvalues of  $JJ^{\top}$  (around  $10^{-2}$ ) that are much smaller than the largest one ( $\geq 1$ ). Observing such a large gap between  $JJ^{\top}$  eigenvalues, it is natural to ask: how do the maximal/minimal eigenvalues affect the DGL attacks?
Figure 6: Same perturbation sizes but different protection effects by different eigenvectors. In (a) and (b), MSEs of DGL attacks are reversely proportional to eigenvalues on the LeNet model. Blue curves are scaled  $1 / \lambda$ . Darker dots indicate smaller MSE (higher risks). Recovered MNIST images associated with different eigenvalues are present on the right.
Figure 5: The distribution of eigenvalues of  $JJ^{\top}$  of two datasets on the LeNet model.
Comparing eigenvectors in defending DGL. We consider a special case of perturbation by letting  $\delta$  be an eigenvector of  $J J^{\top}$ . Then the  $\mathrm{I}^2\mathrm{F}$  will be  $1 / \lambda$  where  $\lambda$  is the corresponding eigenvalue. We conjecture  $1 / \lambda$  could predict the MSE of DGL attacks. To verify the conjecture, we choose 4 eigenvectors with distinct eigenvalues per sample. The results
for the LeNet model are present in Fig. 6. We see that the MSE decreases by  $\lambda$ . For the MNIST dataset, the MSE- $\lambda$  relation is very close to the predicted  $1 / \lambda$ . Though the curve is biased from the ground truth for CIFAR10, we still can use  $1 / \lambda$  to lower bound the recovery error. The bias in CIFAR10 is probably due to the hardness of recovering the more complicated patterns than the digit images. The recovered images in Fig. 6 suggest that even with the same perturbation scale, there exist many bad directions for defense. In the worst case, the image can be fully covered. The observation is an alerting message to the community: protection using random noise may leak private information.
5.2 Privacy Protection Could Be Unfair
Our analysis of the Gaussian perturbation in Eq. (6) indicates that the privacy risk hinges on the Jacobian and therefore the sample. We conjecture that the resulting privacy risk will vary significantly due to the sample dependence. In Fig. 7, we conduct fine-grained experiments to study the variance of privacy protection, when Gaussian perturbation on gradients is used for protection. We use a well-trained model which is thought to be less risky than initial models, for example, in (Balunović et al., 2022). Though the average of MSE implies a reasonable privacy degree as reported in previous literature, the large variance delivers the opposite message that some samples or classes are not that
Figure 7: The sample-wise and class-wise statistics of the DGL MSE on the MNIST dataset, when gradients are perturbed with Gaussian noise of variance  $10^{-3}$ . The purple lines indicate the average values. Large variances are observed among samples and classes. The original (top) and recovered (bottom) images for the well- and poorly-protected classes are depicted on the right side.
safe. In the sense of samples, many samples are more vulnerable than the average. For the classes, some classes are obviously more secure than others. Thus, when the traditional metric focusing on average is used, it may deliver a fake sense of protection unfairly for specific classes or samples.
To understand which kind of samples could be more vulnerable to attack, we look into the Jacobian and ask when  $JJ^{\top}$  will have larger eigenvalues and higher risks in Fig. 7. Consider a poisoning attack, where an attacker aims to maximize the training loss by adding noise  $\delta_{x}$  to the sample  $x$ . Suppose the parameter is updated via  $\theta'(x + \delta_x) = \theta - \nabla_\theta L(x + \delta_x, \theta)$ . Let  $J_\delta = \nabla_x \nabla_\theta L(x_1 + \delta_x, \theta)$  and we can derive the maximal loss amplification on a test sample  $x_1$  when perturbing  $x$  as:
$$
\begin{array}{l} \max  _ {\| \delta_ {x} \| \leq 1} \mathbb {E} _ {x _ {1}} [ L (x _ {1}, \theta^ {\prime} (x + \delta_ {x})) ] \approx \max  _ {\| \delta_ {x} \| \leq 1} \mathbb {E} _ {x _ {1}} [ L (x _ {1}, \theta^ {\prime} (x)) ] + \mathbb {E} _ {x _ {1}} [ \delta_ {x} ^ {\top} \nabla_ {x} L (x _ {1}, \theta^ {\prime} (x)) ] \\ = \max  _ {\left\| \delta_ {x} \right\| \leq 1} \delta_ {x} ^ {\top} J _ {\delta} \mathbb {E} _ {x _ {1}} \left[ \nabla_ {\theta^ {\prime}} L \left(x _ {1}, \theta^ {\prime}\right) \right] \\ = \left\| J _ {\delta} \mathbb {E} _ {x _ {1}} \left[ \nabla_ {\theta^ {\prime}} L \left(x _ {1}, \theta^ {\prime}\right) \right] \right\| \\ \leq \left\| J _ {\delta} \right\| \left\| \mathbb {E} _ {x _ {1}} \left[ \nabla_ {\theta^ {\prime}} L \left(x _ {1}, \theta^ {\prime}\right) \right] \right\|. \\ \end{array}
$$
As shown by the inequality, the sample with large  $\| J_{\delta} \|$  may significantly bias the training after mild perturbation  $\delta_x$ . If Assumption 3.4 holds, then  $\| J \|$  can be associated with  $\| J_{\delta} \|$  by  $| \| J \| - \| J_{\delta} \| | \leq \mu_J$ . Now we can connect the privacy vulnerability to the data poisoning. With Eq. (6), samples that suffer high privacy risks due to large  $\| J \|$  could also be influential in training and can be easily used to poison training.
5.3 Model Initialization Matters
Since Jocabian leans on the parameters, we hypothesize that the way we initialize a model can also impact privacy. Like previous work (Sun et al., 2020; Balunović et al., 2022; Zhu et al., 2019), we are interested in the privacy risks when a model is randomly initialized without training. Unlike previous work, we further ask which initialization strategy could favor privacy protection under the same Gaussian perturbation on gradients.
Case Study: One-layer network. A one-layer network with nonlinear activation  $\sigma (\cdot)$  is a simple yet sufficient model for analyzing initialization. Let  $L(x,\theta) = \frac{1}{2}\left\| \sigma (\theta^{\top}x) - b\right\|^{2}$ , where  $b$  is a constant. Denote the product  $\theta^{\top}x$  as  $a$ . The Jacobian is given by:
(b) CIFAR10
Figure 8: Different initialization strategies could result in distinct MSEs.
$$
J = \frac {\partial^ {2} L}{\partial x \partial \theta} = \frac {\partial \sigma}{\partial a} \left\{\left(\sigma \left(\theta^ {\top} x\right) - b\right) I + x \theta^ {\top} \right\}.
$$
(1) Apparently, a well-trained  $\theta$  with
$(\sigma(\theta^{\top}x) - b) = 0$  will cause  $J$  to be a rank-one matrix. Note that the gradient will be zero in such a case and no longer provide any information about the private sample. Consistent with our theoretical results, where the inverse Jacobian makes  $\mathcal{I}$  infinite, the gradient will be perfectly private at any perturbation. (2) If  $\sigma(\theta^{\top}x) - b$  is non-zero, the singular values of  $J$  will depend on how close  $\mathbb{E}[\sigma(\theta^{\top}x)]$  is to  $b$ . If  $\mathbb{E}[\sigma(\theta^{\top}x)]$  approaches  $b$  at initialization, then  $J$  will be approximately singular, and therefore the initialization enhances privacy preservation.
Comparing initialization. To evaluate the effect of initialization on the inversion, we conduct experiments of the LeNet model on MNIST and CIFAR10 datasets with DGL and GS attacks. We add Gaussian noise to the gradients, which implies an expected MSE should be proportional to  $\mathbb{E}[\mathcal{I}^2] = \sum_i \lambda_i^{-1}$  as proved in Eq. (6). Here, we evaluate how the initialization will change eigenvalues of  $JJ^{\top}$  and thus change the MSE of DGL. Four commonly used initialization techniques are considered here: uniform distribution in range  $[-0.5, 0.5]$ , normal distribution with zero mean and variance 0.5, kaiming (He et al., 2015b) and xavier (Glorot and Bengio, 2010) initialization. For each initialization, the same 10 samples are used to conduct the attack three times. The results are shown in Fig. 8. We observe a significant gap between initialization mechanisms. Using uniform initialization cast serious risks of leaking privacy under the same Gaussian defense. Though not as significant as uniform initialization, the normal initialization is riskier than rest two techniques. kaiming and xavier methods can favor convergence in deep learning and here we show that they are also preferred for privacy. A potential reason is that the two methods can better normalize the activations to promote the Jacobian singularity.
In parallel with our work, Wang et al. (2023a) also finds that the initialization is impactful on inversion attacks. Despite their analysis being based on a different metric, the layer-wise variance of the model weights, their work and ours have a consistent conclusion that models initialized with a uniform distribution face higher privacy risks than that with a kaiming distribution
In this paper, we introduce a novel way to use the influence functions for analyzing Deep Gradient Leakage (DGL). We propose a new and efficient approximation of DGL called the Inversion Influence Function  $(\mathrm{I}^2\mathrm{F})$ . By utilizing this tool, we gain valuable insights into the occurrence and mechanisms of DGL, which can greatly help the future development of effective defense methods.
Limitations. Our work may be limited by some assumptions and approximations. First, we worked on the worst-case scenario where a strong attack conducts perfect inversion attacks. In practice, such an assumption can be strong, especially for highly complicated deep networks. The gap between existing attacks and perfect attacks sometimes leads to a notable bias. However, we note that recent years witnessed many techniques that significantly improved attacking capability (Geiping et al., 2020; Jeon et al., 2021; Zhao et al., 2020), and our work is valuable to bound the risks when the attacks get even stronger over time. Second, similar to the traditional influence function,  $\mathrm{I}^2\mathrm{F}$  can be less accurate and suffers from large variance in extremely non-convex loss functions. Advanced linearization techniques (Bae et al., 2022) can be helpful in improving the accuracy of influence. Then extending our analysis to bigger foundation models may bring intriguing insights into the scaling law of privacy.
Future Directions. As the first attempt at influence function in DGL, our method can serve multiple purposes to benefit future research. For example, our metric can be used to efficiently examine the privacy breach before sending gradients to third parties. Since  $\mathrm{I}^2\mathrm{F}$  provides an efficient evaluation of the MSE, it may be directly optimized in conjunction with the loss of main tasks. Such joint optimization could bring in the explicit trade-off between utility and privacy in time. In comparison, traditional arts like differential privacy are complicated by tuning the privacy parameter for the trade-off. Furthermore, we envision that many techniques can be adopted to further enhance the analysis. For example, unrolling-based analysis leverages the iterative derivatives in the DGL to uncover the effectiveness of gradient perturbations (Pruthi et al., 2020).
Broader Impacts. Data privacy has been a long-term challenge in machine learning. Our work provides a fundamental tool to diagnose privacy breaches in the gradients of deep networks. Understanding when and how privacy leakage happens can essentially help the development of defenses. For example, it can be used for designing stronger attacks, which leads to improved defense mechanisms and ultimately benefit the privacy and security of machine learning.
This research was supported by the National Science Foundation (IIS-2212174, IIS-1749940), National Institute of Aging (IRF1AG072449), and the Office of Naval Research (N00014- 20-1- 2382).
A.1 Other Efficient Evaluation Techniques
The techniques for efficiently evaluating implicit gradients can be referred to (Koh and Liang, 2017; Grazzi et al., 2020). As computing the inverse second-order derivatives is the most computation-intensive operation, we will focus on it. Here, we briefly summarize two supplementary techniques introduced in Section 3.1.
Conjugate gradient. In Section 3.1, we use the trick of least square to compute the  $(JJ^{\top})^{-1}J\delta$ . When  $JJ^{\top} \succ 0$ , we can solve the least square problem by the conjugate gradient (CG) method, which only needs  $\mathcal{O}(d)$  time to converge. However, the algorithm will be unstable when the matrix  $JJ^{\top}$  is ill-conditioned. As observed in Fig. 5, the  $JJ^{\top}$  is likely to be ill-conditioned for deep networks. But here we provide this alternative for linear models such that  $J$  can be evaluated faster.
Neumann series. We can leverage the Neumann series to compute the matrix inverse. By the Neumann series, we have  $(JJ^{\top})^{-1}J\delta = \lim_{t\to \infty}\sum_{i = 0}^{t}(I - JJ^{\top})^{i}J\delta$ . Let  $s_t\triangleq \sum_{i = 0}^t (I - JJ^\top)^i J\delta$  and  $s_0\triangleq J\delta$ . Then the computation can be done by iteration  $s_{t + 1} = (I - JJ^{\top})s_{t} + J\delta$  which only includes Jacobian-vector products.
B.1 Proof of the Approximation by Implicit Gradients
Here, we provide the proof for  $\frac{\partial G_r(g_0)}{\partial g_0} = (JJ^\top)^{-1}J$ . Recall Eq. (1) as
$$
x ^ {*} = G _ {r} (g) = \underset {x} {\arg \min } L _ {I} (x; g) \triangleq \| \nabla_ {\theta} L (x, \theta) - g \| ^ {2}. \tag {8}
$$
The stationary condition of the minimization gives
$$
\frac {\partial L _ {I} \left(x ^ {*} ; g\right)}{\partial x ^ {*}} = 0.
$$
Given a small perturbation  $\Delta_g \to 0$  on the gradient, we can estimate corresponding perturbation  $\Delta_{x^*} \to 0$  as a function of  $\Delta_g$ . Thus, we can approximate  $\frac{\partial x^*}{\partial g}$  by  $\frac{\partial \Delta_{x^*}}{\partial \Delta_g}$ . Use Taylor expansion to show approximately
$$
\frac {\partial L _ {I} \left(x ^ {*} ; g\right)}{\partial x ^ {*}} + \frac {\partial^ {2} L _ {I} \left(x ^ {*} ; g\right)}{\partial g \partial x ^ {*}} \Delta_ {g} + \frac {\partial^ {2} L _ {I} \left(x ^ {*} ; g\right)}{\partial x ^ {* 2}} \Delta_ {x ^ {*}} \approx 0
$$
$$
\Delta_ {x ^ {*}} \approx - \left(\frac {\partial^ {2} L _ {I} (x ^ {*} ; g)}{\partial x ^ {* 2}}\right) ^ {- 1} \frac {\partial L _ {I} (x ^ {*} ; g)}{\partial x ^ {*}} \Delta_ {g}
$$
$$
\frac {\partial x _ {g} ^ {*}}{\partial g} \approx - \left(\frac {\partial^ {2} L _ {I} \left(x ^ {*} ; g\right)}{\partial x ^ {* 2}}\right) ^ {- 1} \frac {\partial^ {2} L _ {I} \left(x ^ {*} ; g\right)}{\partial g \partial x ^ {*}} \tag {9}
$$
where we drop higher-order perturbations. The above derivations can be rigorously proved using the Implicit Function Theorem. Since  $\frac{\partial L_I(x^*;g)}{\partial x^*} = 2(\nabla_\theta L(x^*,\theta) - g)\nabla_x\nabla_\theta L(x^*,\theta)$ , we can derive
$$
\frac {\partial^ {2} L _ {I} \left(x ^ {*} ; g\right)}{\partial g \partial x ^ {*}} = - 2 \nabla_ {x} \nabla_ {\theta} L \left(x ^ {*}, \theta\right)
$$
$$
\frac {\partial^ {2} L _ {I} \left(x ^ {*} ; g\right)}{\partial x ^ {* 2}} = 2 \left(\nabla_ {\theta} L \left(x ^ {*}, \theta\right) - g\right) \nabla_ {x} ^ {2} \nabla_ {\theta} L \left(x ^ {*}, \theta\right) + 2 \nabla_ {x} \nabla_ {\theta} L \left(x ^ {*}, \theta\right) \left(\nabla_ {x} \nabla_ {\theta} L \left(x ^ {*}, \theta\right)\right) ^ {\top}.
$$
As  $x_0 = x^* = G_r(g_0)$  and  $g_0 = \nabla_\theta L(x^*,\theta)$ , we can substitute them to obtain
$$
\frac {\partial^ {2} L _ {I} \left(x _ {0} ; g\right)}{\partial g \partial x _ {0}} = - 2 \nabla_ {x} \nabla_ {\theta} L \left(x _ {0}, \theta\right) := - 2 J \left(x ^ {*} \left(g _ {0}\right), \theta\right), \tag {10}
$$
$$
\frac {\partial^ {2} L _ {I} \left(x _ {0} ; g\right)}{\partial x _ {0} ^ {2}} = 2 \left(g _ {0} - g\right) \nabla_ {x} ^ {2} \nabla_ {\theta} L \left(x _ {0}, \theta\right) + 2 J J ^ {\top}. \tag {11}
$$
Let  $g = g_{0}$ . Combine Eqs. (9) to (11) to get
$$
\frac {\partial G _ {r} (g _ {0})}{\partial g _ {0}} = \left(J J ^ {\top}\right) ^ {- 1} J.
$$
Before we prove our main theorem, we prove several essential lemmas as below.
Lemma B.1.  $\begin{array}{r}\left\| \nabla_{x}\left\| \nabla_{\theta}L(x,\theta)\right\|^{2} - \nabla_{x}\left\| \nabla_{\theta}L(x^{\prime},\theta)\right\|^{2}\right\| \leq \end{array}$ $(\mu_L\| \nabla_x\nabla_\theta L(x,\theta)\| +$ $\mu_J\| \nabla_\theta L(x',\theta)\|)\| x - x'\|$