SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In this section, we provide a computational guarantee to Algorithm 1 in terms of the number of gradient evaluations on individual loss functions to compute the approximate minimizer for achieving (up to a constant) the information-theoretical limit.
Let  $f(\theta) \coloneqq \sum_{i} \ell_i(\theta) + \frac{\lambda}{2} \| \theta \|^2 + b^T \theta$ , i.e., the perturbed and regularized objective function used by ObjPert. Let  $\theta^{**}$  be the output returned by the blackbox algorithm  $\theta^A(\cdot)$  described in Section E. Note the deviation from the notation used in the previous section.
Iyengar et al. (2019) proposed a procedure that keeps checking the gradients in an iterative optimization algorithm and stops when the gradient is smaller than  $\tau$ . This always ensures that  $\| \nabla f(\theta^{**}) \| \leq \tau$ .
And using tools from the next section, it can be proven that it implies that  $\| \theta^{**} - \theta^*\|_2 \leq \tau / \lambda$  as was previously stated. But how many iterations it takes for this to happen for specific algorithms was not explicitly considered.
F.1 Tools from convex optimization
We will need a few tools from convex optimization.
Firstly, under our assumption that  $\ell_i$  is  $\beta$ -smooth,  $f$  is  $n\beta + \lambda$ -smooth and  $\lambda$ -strongly convex. Let  $L \coloneqq n\beta + \lambda$  as a shorthand.
By  $L$ -smoothness (gradient Lipschitzness), and the optimality of  $\theta^{*}$ , we have that for any  $\theta$
$$
\| \nabla f (\theta) \| = \| \nabla f (\theta) - \nabla f \left(\theta^ {*}\right) \| \leq L \| \theta - \theta^ {*} \| _ {2}. \tag {F.1}
$$
By  $\lambda$ -strong convexity, we get
$$
f (\theta) - f ^ {*} \geq \frac {\lambda}{2} \| \theta - \theta^ {*} \| ^ {2} \geq \frac {\lambda}{2 L ^ {2}} \| \nabla f (\theta) \| ^ {2} \tag {F.2}
$$
By strong convexity also implies that
$$
\| \nabla f (\theta) \| \geq \lambda \| \theta - \theta^ {*} \| \tag {F.3}
$$
which is the quantity used to establish the global sensitivity of  $q(D, \theta^{*})$  as we talked about earlier.
(F.2) and (F.3) sandwich  $\| \theta -\theta^{*}\|$  in between by
$$
\frac {\| \nabla f (\theta) \|}{L} \leq \| \theta - \theta^ {*} \| \leq \frac {\| \nabla f (\theta) \|}{\lambda}.
$$
F.2 Computational bounds for Stopping at small gradient
(F.1) and (F.2) together provides bounds for  $\| \nabla f(\theta)\|$  using either objective function or argument convergence (in square  $\ell_2$ .)
$$
\left\| \nabla f (\theta) \right\| ^ {2} \leq \min  \left\{\frac {2 L ^ {2}}{\lambda} (f (\theta) - f ^ {*}), L ^ {2} \| \theta - \theta^ {*} \| ^ {2} \right\}.
$$
Standard convergence results are often parameterized in terms of either suboptimality  $f(\theta) - f^{*}$  or argument  $\| \theta - \theta^{*} \|^{2}$ . In the following we instantiate specific convergence bounds for deriving computation guarantees.
Gradient Descent. If we run gradient descent with learning rate  $1 / L$  for  $T$  iterations from  $\theta_0$ , then
$$
\left\| \theta_ {T} - \theta^ {*} \right\| ^ {2} \leq \left(1 - \frac {\lambda}{L}\right) ^ {T} \left\| \theta_ {0} - \theta^ {*} \right\| ^ {2}
$$
$$
\left\| \nabla f \left(\theta_ {T}\right) \right\| ^ {2} \leq L ^ {2} \left(1 - \frac {\lambda}{L}\right) ^ {T} \left\| \theta_ {0} - \theta^ {*} \right\| ^ {2}
$$
This happens deterministically (with no randomness, or failure probability).
One may ask why are we not running a fixed number of iterations and directly applying the bound to  $\| \theta_T - \theta^*\|$  in order to control the  $\ell_2$  sensitivity. That works fine, except that we have an unconstrained problem and  $\theta^*$  can be anywhere, thus there might not be a fixed parameter  $T$  to provide a required bound for all input  $\theta^*$ . We also do not know where  $\theta^*$  is during the actual execution of the algorithm and thus cannot compute  $\| \theta_0 - \theta^*\|$  directly.
The "gradient-norm check" as a stopping condition from Iyengar et al. (2019) is nice because it always ensures DP for any  $\theta^{*}$  (at a price of sometimes running for a bit longer).
To ensure  $\| \nabla f(\theta)\| \leq \gamma$ , the number of iterations
$$
T = \frac {\log (\frac {L ^ {2} \| \theta_ {0} - \theta^ {*} \| ^ {2}}{\gamma^ {2}})}{\log (1 + \frac {\lambda}{L - \lambda})} \leq \frac {2 (L - \lambda)}{\lambda} \log (\frac {L ^ {2} \| \theta_ {0} - \theta^ {*} \| ^ {2}}{\gamma^ {2}}) = \frac {2 n \beta}{\lambda} \log (\frac {(n \beta + \lambda) ^ {2} \| \theta_ {0} - \theta^ {*} \| ^ {2}}{\gamma^ {2}})
$$
Since each gradient computation requires  $n$  incremental gradient evaluation, under the regime that  $\lambda$  is independent of  $n$ , under the choice that  $\lambda = 1 / \epsilon$  independent to  $n$  from the standard calibration, the total number of is therefore  $O(n^{2}\log n)$  for achieving  $\gamma \leq n^{-v}$  for any constant  $v > 0$ .
The quadratic runtime is not ideal, but it can be improved using accelerated gradient descent which gives a convergence bound of
$$
f (\theta_ {T}) - f ^ {*} \leq (1 - \sqrt {\frac {\lambda}{L}}) ^ {T} \| \theta_ {0} - \theta^ {*} \| ^ {2}.
$$
This would imply a computational guarantee of  $O(n^{1.5}\log n)$ . The overall computation bound depends on  $\| \theta^* \|$  which is random (due to objective perturbation). The dependence on  $\| \theta^* \|$  is only logarithmic though.
Finite Sum and SAG. The result can be further improved if we use stochastic gradient methods. However, the sublinear convergence of the standard SGD or its averaged version makes the application of the above conversion rules somewhat challenging.
By taking advantage of the finite sum structure of  $f(\theta)$  one can obtain faster convergence.
First of all, the finite sum structure says that  $f(\theta) = \sum_{i=1}^{n} f_i(\theta)$ . In our case, we can split the regularization and linear perturbation to the  $n$  data points, i.e.,
$$
f _ {i} (\theta) = \ell_ {i} (\theta) + \frac {\lambda}{2 n} \| \theta \| ^ {2} + \frac {b ^ {T} \theta}{n}.
$$
Check that it satisfies  $\beta +\lambda /n$  smoothness.
There is a long list of methods that satisfy the faster convergence for finite sum problems, e.g., SAG, SVRG, SAGA, SARAH and so on (see, e.g., Nguyen et al., 2022, for a recent survey). Specifically, Stochastic Averaged Gradient (Schmidt et al., 2017) (and similarly others with slightly different parameters) satisfies
$$
\mathbb {E} \left[ f (\boldsymbol {\theta} ^ {T}) - f ^ {*} \right] \leq (1 - \min \{\frac {\lambda}{1 6 L}, \frac {1}{8 n} \}) ^ {T} \cdot (\frac {3 n}{2} (f (\boldsymbol {\theta} _ {0}) - f ^ {*}) + 4 L \| \boldsymbol {\theta} _ {0} - \boldsymbol {\theta} ^ {*} \| ^ {2}).
$$
Therefore, by (F.2), we have
$$
\mathbb {E} \left[ \| \nabla f (\theta_ {T}) \| ^ {2} \right] \leq (1 - \min \{\frac {\lambda}{1 6 L}, \frac {1}{8 n} \}) ^ {T} \cdot \frac {L ^ {2}}{\lambda} (\frac {3 n}{2} (f (\theta_ {0}) - f ^ {*}) + 4 L \| \theta_ {0} - \theta^ {*} \| ^ {2}).
$$
Note that each iteration costs just one incremental gradient evaluation, so to ensure  $\mathbb{E}[\| \nabla f(\theta_T)\|^2 ]\leq \gamma^2$ , the computational complexity is on the order of
$$
\max  \left\{n, \frac {L}{\lambda} \right\} \log \left(\frac {n L \max  \left\{f \left(\theta_ {0}\right) - f ^ {*} , \| \theta_ {0} - \theta^ {*} \| ^ {2} \right\}}{\lambda \gamma}\right)
$$
This is  $O(n \log n)$  runtime to any  $\gamma = n^{-s}$  for a constant  $s > 0$ .
On the other hand, the main difference from the gradient descent result is that we only get convergence in expectation. By Markov's inequality
$$
\mathbb {P} \left[ \| \nabla f (\theta_ {T}) \| ^ {2} > \gamma^ {2} \right] \leq \frac {(1 - \min \{\frac {\lambda}{1 6 L} , \frac {1}{8 n} \}) ^ {T} \cdot \frac {L ^ {2}}{\lambda} (\frac {3 n}{2} (f (\theta_ {0}) - f ^ {*}) + 4 L \| \theta_ {0} - \theta^ {*} \| ^ {2})}{\gamma^ {2}} := \delta ,
$$
which implies high probability convergence naturally.
Theorem F.1. Assume  $\lambda \geq \beta$ . The algorithm that runs SAG and checks the stopping condition  $\| \nabla f(\theta_T) \| \leq \gamma$  after every  $n$  iteration will terminate with probability at least  $1 - \delta$  in less than
$$
C \max  \left\{n, \frac {n \beta}{\lambda} \right\} \log \left(\frac {n \beta \max  \left\{\| \theta_ {0} - \theta^ {*} \| , (f (\theta_ {0}) - f ^ {*}) \right\}}{\gamma \delta}\right)
$$
incremental gradient evaluations, where  $C$  is a universal constant.
How to set  $\gamma$  to achieve information-theoretic limit? The lower bounds for convex and smooth losses in differentially private ERM are well-known (Bassily et al., 2014) and it is known that among GLMs,  $\theta^{*}$  from ObjPert achieves the lower bound with appropriate choices of  $\lambda, \sigma$ . Notably,  $\lambda \asymp d / \epsilon$  for achieving an  $(\epsilon, \delta)$ -DP.
$$
\mathbb {E} [ \sum_ {i} \ell_ {i} (\theta^ {*}) ] - \min _ {\theta} \sum_ {i} \ell_ {i} (\theta) \leq \operatorname {M i n i m a x E x c e s s E m p i r i a l R i s k}
$$
Let  $\hat{\theta} = \theta_T + N(0, \frac{\gamma^2}{2\lambda^2\rho} I)$  be the final output.
By the  $nG$  Lipschitzness of  $\sum_{i}\ell_{i}$ , with high probability over the Gaussian mechanism, we have that
$$
\mathbb {E} [ \sum_ {i} \ell_ {i} (\hat {\theta}) ] - \sum_ {i} \ell_ {i} (\theta^ {*}) \leq n G (\| \theta_ {T} - \theta^ {*} \| + \| \hat {\theta} - \theta_ {T} \|) \leq n G \gamma (1 + \frac {\sqrt {\frac {d \log d}{\rho}}}{\lambda})
$$
where  $\rho$  is the zCDP parameter for the Gaussian mechanism chosen to match the large  $\alpha$  part of the ObjectivePerturbation's RDP bound, which increases the overall RDP by  $\alpha\rho$ .
Thus, it suffices to take  $\gamma = \mathrm{MinimaxExcessEmpiricalRisk} / (nG(1 + \frac{\sqrt{d\log d / \rho}}{\lambda}))$
To conclude, the above results imply that the computationally efficient objective perturbation achieves the optimal rate under the same RDP guarantee with an algorithm that terminates in  $O(n \log n)$  time with high probability.
G The GLM Bug
G.1 Discussion
Limiting our main results to generalized linear models might appear restrictive — but we argue that the GLM assumption is not specific to our paper, but rather has been lurking in the objective perturbation literature for some time now.
Let's first take a look at Section 3.3.2 of Chaudhuri et al. (2011): Lemma 10 requires that the matrix  $E$  have rank at most 2, but this is not necessarily true without assuming GLM structure. This is used to bound the determinant of the Jacobian, and corresponds to the first term of our bound in Theorem 3.2.
It is a similar story for bounding the log ratio / difference between the noise vector densities under neighboring datasets, corresponding to the second and third terms of our bound in Theorem 3.2. Let's also revisit this line from the proof of Lemma 17 of the Kifer et al. (2012) paper: "Note that  $\Gamma$  is independent of the noise vector." This is not true without assuming GLM structure! (In their proof,  $\Gamma$  is the difference between the noise vectors under neighboring distributions. From first-order conditions at the minimizer of the perturbed objective, we can see that  $\Gamma = \nabla \ell (\theta^P)$ , where  $\theta^P$  is a function of the noise vector  $b$ .
In fact, to our knowledge, Iyengar et al. (2019) was the first work to acknowledge the GLM assumption on objective perturbation. But their privacy proof also fails to handle the dependence on the noise vector! In Theorems 3.2 and 3.1, we have included a careful analysis including a discussion on how the GLM assumption removes this dependence.
G.2 RDP bound for non-GLMs
In this section we generalize the RDP bound for objective perturbation to a general class of smooth convex losses.
We first state the following bound from Theorem 6
Theorem G.1.
$$
\epsilon_ {1} (\hat {\theta} ^ {P}, D, D _ {\pm z}) = \left| - \log \prod_ {j = 1} ^ {d} \Big (1 \mp \mu_ {j} \Big) + \frac {1}{2 \sigma^ {2}} | | \nabla \ell (\hat {\theta} ^ {P}; z) | | _ {2} ^ {2} \pm \frac {1}{\sigma^ {2}} \nabla J (\hat {\theta} ^ {P}; D) ^ {T} \nabla \ell (\hat {\theta} ^ {P}; z) \right|,
$$
where  $\mu_{j} = \lambda_{j}u_{j}^{T}\Big(\nabla b(\hat{\theta}^{P};D)\mp \sum_{k = 1}^{j - 1}\lambda_{k}u_{k}u_{k}^{T}\Big)^{-1}u_{j}$  according to the eigendecomposition  $\nabla^2\ell (\theta ;z) = \sum_{k = 1}^d\lambda_ku_ku_k^T$
Theorem G.2 (RDP bound for non-GLMS.). Let  $||\nabla \ell(\theta;z)||_2 \leq L$  and  $\nabla^2 \ell(\theta;z) \prec \beta I_d$  for all  $\theta \in \Theta$  and  $z \in \mathcal{X} \times \mathcal{Y}$ . Objective perturbation satisfies  $(\alpha, \epsilon)$ -RDP for any  $\alpha > 1$  with
$$
\epsilon = - d \log \left(1 - \frac {\beta}{\lambda}\right) + \frac {L}{2 \sigma^ {2}} + \frac {1}{\alpha - 1} \log \mathbb {E} _ {Z \sim \chi_ {d}} \left[ e ^ {(\alpha - 1) \frac {L ^ {2}}{\sigma^ {2}} Z} \right],
$$
where  $Z \sim \chi_d$  if  $Z = \sqrt{\sum_{i=1}^{d} X_i^2}$  and  $X_i \sim \mathcal{N}(0,1)$  for all  $i \in [d]$ .
Proof. The above follows from Theorem 6 in (Redberg & Wang, 2021) applied to the analysis of Theorem 3.2. We find that
$$
\left| - \log \prod_ {j = 1} ^ {d} (1 \mp \mu_ {j}) \right| = \left| - \sum_ {j = 1} ^ {d} \log (1 \mp u _ {j}) \right| \leq - \sum_ {j = 1} ^ {d} \log (1 - u _ {j}) \leq - d \log \left(1 - \frac {\beta}{\lambda}\right).
$$
We now need to bound
$$
\frac {1}{\alpha - 1} \log \mathbb {E} \left[ e ^ {\left(\alpha - 1\right) \frac {1}{\sigma^ {2}} \nabla J \left(\theta^ {P}\right) ^ {T} \nabla \ell \left(\theta^ {P}\right)} \right].
$$
Recall that  $b \sim \mathcal{N}(0, \sigma^2 I_d)$ . Using the first-order condition  $\nabla J(\theta^P) = -b$  and the Cauchy-Schwarz inequality, we have
$$
- b ^ {T} \nabla \ell (\theta^ {P}) \leq | - b ^ {T} \nabla \ell (\theta^ {P}) | \leq L | | b | | _ {2}.
$$
$$
\frac {1}{\alpha - 1} \log \mathbb {E} \left[ e ^ {(\alpha - 1) \frac {1}{\sigma^ {2}} \nabla J (\theta^ {P}) ^ {T} \nabla \ell (\theta^ {P})} \right] \leq \frac {1}{\alpha - 1} \log \mathbb {E} _ {Z \sim \chi_ {d}} \left[ e ^ {(\alpha - 1) \frac {L}{\sigma^ {2}} Z} \right].
$$
H Excess Empirical Risk of Algorithm 1
Our goal in this section is to find a bound on the excess empirical risk:
$$
\mathbb {E} \left[ \mathcal {L} \left(\tilde {\theta} ^ {P}; Z\right) \right] - \mathcal {L} \left(\theta^ {*}; Z\right).
$$
Theorem H.1. Let  $\tilde{\theta}^P$  be the output of Algorithm 1 and  $\theta^{*} = \arg \min \mathcal{L}(\theta)$  the minimizer of the loss function  $\mathcal{L}(\theta) = \sum_{i=1}^{n} \ell(\theta; z_i)$ . Denote  $||\mathcal{X}||$  as the diameter of the set  $\mathcal{X}$ . We have
$$
\mathbb {E} \left[ \mathcal {L} (\tilde {\theta} ^ {P}; Z) \right] - \mathcal {L} (\theta^ {*}; Z) \leq n L \left(\frac {\tau}{\lambda} + \sigma_ {o u t} \sqrt {d}\right) + \frac {(n \beta | | \mathcal {X} | | _ {2} ^ {2} + \lambda) d \sigma^ {2}}{2 \lambda^ {2}} + \frac {\lambda}{2} | | \theta^ {*} | | _ {2} ^ {2}.
$$
Proof. Following the proof of Theorem 2 from Iyengar et al. (2019) (itself adapted from Kifer et al. (2012)), we write
$$
\mathcal {L} (\tilde {\theta} ^ {P}) - \mathcal {L} (\theta^ {*}) = \left(\mathcal {L} (\tilde {\theta} ^ {P}) - \mathcal {L} (\theta^ {P})\right) + \left(\mathcal {L} (\theta^ {P}) - \mathcal {L} (\theta^ {*})\right).
$$
By the  $\lambda$ -strong convexity of  $\mathcal{L}^P$ , for any  $\tilde{\theta}, \theta^*$  we have
$$
\left(\nabla \mathcal {L} ^ {P} (\tilde {\theta}) - \nabla \mathcal {L} ^ {P} \left(\theta^ {P}\right)\right) ^ {T} \left(\tilde {\theta} - \theta^ {P}\right) \geq \lambda | | \tilde {\theta} - \theta^ {P} | | _ {2} ^ {2}.
$$
By first-order conditions,  $\nabla \mathcal{L}^P (\theta^P) = 0$ . Applying the Cauchy-Schwarz inequality along with our stopping criteria on the gradient norm, we then have
$$
\left| \left| \tilde {\theta} - \theta^ {P} \right| \right| _ {2} \leq \frac {1}{\lambda} \left| \left| \nabla \mathcal {L} ^ {P} (\tilde {\theta}) \right| \right| _ {2} \leq \frac {\tau}{\lambda}.
$$
Let  $b_{2} \sim \mathcal{N}(0, \sigma_{out}^{2} I_{d})$ . Because  $\mathcal{L}$  is  $nL$ -Lipschitz continuous, we have
$$
\begin{array}{l} \left(\mathcal {L} \left(\tilde {\theta} ^ {P}\right) - \mathcal {L} \left(\theta^ {P}\right)\right) \leq n L | | \tilde {\theta} ^ {P} - \theta^ {P} | | _ {2} \\ = n L | | \tilde {\theta} + b _ {2} - \theta^ {P} | | _ {2} \\ \leq n L \left(\left| \left| \tilde {\theta} - \theta^ {P} \right| \right| _ {2} + \left| \left| b _ {2} \right| \right| _ {2}\right) \\ \leq n L \left(\frac {\tau}{\lambda} + | | b _ {2} | | _ {2}\right). \\ \end{array}
$$
By Lemma K.10,
$$
\mathbb {E} \left[ n L \left(\frac {\tau}{\lambda} + | | b _ {2} | | _ {2}\right) \right] \leq n L \left(\frac {\tau}{\lambda} + \sigma_ {o u t} \sqrt {d}\right).
$$
To bound the expectation of  $\mathcal{L}(\theta^P) - \mathcal{L}(\theta^*)$ , we can write
$$
\mathcal {L} \left(\theta^ {P}\right) - \mathcal {L} \left(\theta^ {*}\right) = \left(\mathcal {L} \left(\theta^ {P}\right) - \mathcal {L} _ {\lambda} \left(\theta^ {P}\right)\right) + \left(\mathcal {L} _ {\lambda} \left(\theta^ {P}\right) - \mathcal {L} _ {\lambda} \left(\theta_ {\lambda} ^ {*}\right)\right) + \left(\mathcal {L} _ {\lambda} \left(\theta_ {\lambda} ^ {*}\right) - \mathcal {L} _ {\lambda} \left(\theta^ {*}\right)\right) + \left(\mathcal {L} _ {\lambda} \left(\theta^ {*}\right) - \mathcal {L} \left(\theta^ {*}\right)\right).
$$
$$
\mathcal {L} \left(\theta^ {P}\right) - \mathcal {L} _ {\lambda} \left(\theta^ {P}\right) = - \frac {\lambda}{2} | | \theta^ {P} | | _ {2} ^ {2} \leq 0,
$$
and that by optimality  $\left(\theta_{\lambda}^{*} = \operatorname *{arg  min}_{\theta \in \mathbb{R}^{d}}\mathcal{L}_{\lambda}(\theta)\right)$  , we have
$$
\mathcal {L} _ {\lambda} \left(\theta_ {\lambda} ^ {*}\right) - \mathcal {L} _ {\lambda} \left(\theta^ {*}\right) \leq 0.
$$
Observe also that
$$
\mathcal {L} _ {\lambda} \left(\theta^ {*}\right) - \mathcal {L} \left(\theta^ {*}\right) = \frac {\lambda}{2} | | \theta^ {*} | | _ {2} ^ {2}.
$$
$$
\mathcal {L} \left(\theta^ {P}\right) - \mathcal {L} \left(\theta^ {*}\right) \leq \mathcal {L} _ {\lambda} \left(\theta^ {P}\right) - \mathcal {L} _ {\lambda} \left(\theta_ {\lambda} ^ {*}\right) + \frac {\lambda}{2} \| \theta^ {*} \| _ {2} ^ {2}.
$$
By Taylor's Theorem, for some  $\theta^{\prime}\in [\theta^{P},\theta_{\lambda}^{*}]$  we can write
$$
\begin{array}{l} \mathcal {L} _ {\lambda} \left(\theta^ {P}\right) - \mathcal {L} _ {\lambda} \left(\theta_ {\lambda} ^ {*}\right) = \nabla \mathcal {L} _ {\lambda} \left(\theta_ {\lambda} ^ {*}\right) \left(\theta^ {P} - \theta_ {\lambda} ^ {*}\right) ^ {T} + \frac {1}{2} | | \theta^ {P} - \theta_ {\lambda} ^ {*} | | _ {\nabla^ {2} \mathcal {L} _ {\lambda} \left(\theta^ {\prime}\right)} ^ {2} \\ = \frac {1}{2} \left\| \theta^ {P} - \theta_ {\lambda} ^ {*} \right\| _ {\nabla^ {2} \mathcal {L} _ {\lambda} \left(\theta^ {\prime}\right)} ^ {2}. \tag {H.1} \\ \end{array}
$$
The last equality is due to optimality conditions. We can then bound
$$
\begin{array}{l} \frac {1}{2} | | \theta^ {P} - \theta_ {\lambda} ^ {*} | | _ {\nabla^ {2} \mathcal {L} _ {\lambda} (\theta^ {\prime})} ^ {2} \leq \frac {1}{2} | | \nabla^ {2} \mathcal {L} _ {\lambda} (\tilde {\theta}) | | _ {\mathrm {o p}} | | \theta^ {P} - \theta_ {\lambda} ^ {*} | | _ {2} ^ {2} \\ \leq \frac {1}{2} \left(n \beta | | \mathcal {X} | | _ {2} ^ {2} + \lambda\right) \frac {| | b | | _ {2} ^ {2}}{\lambda^ {2}}, \\ \end{array}
$$
where  $b\sim \mathcal{N}(0,\sigma^2 I_d)$  . Then taking the expectation,
$$
\mathbb {E} \left[ \mathcal {L} _ {\lambda} \left(\theta^ {P}\right) \right] - \mathbb {E} \left[ \mathcal {L} _ {\lambda} \left(\theta_ {\lambda} ^ {*}\right) \right] \leq \frac {(n \beta | | \mathcal {X} | | _ {2} ^ {2} + \lambda) d \sigma^ {2}}{2 \lambda^ {2}}.
$$
Altogether we then have
$$
\mathbb {E} \left[ \mathcal {L} \left(\tilde {\theta} ^ {P}; Z\right) \right] - \mathcal {L} \left(\theta^ {*}; Z\right) \leq n L \left(\frac {\tau}{\lambda} + \sigma_ {o u t} \sqrt {d}\right) + \frac {\left(n \beta \| \mathcal {X} \| _ {2} ^ {2} + \lambda\right) d \sigma^ {2}}{2 \lambda^ {2}} + \frac {\lambda}{2} \| \theta^ {*} \| _ {2} ^ {2}.
$$
The optimal choice of  $\lambda$  would then be  $\lambda \asymp \frac{dL}{||\theta^{*}||_{2\epsilon}}$ . The optimal choice of  $\tau$  is discussed in Section F.
H.1 Generalized Linear Model
With some additional assumptions and restrictions, we can get a tighter bound on  $\mathbb{E}\left[\mathcal{L}_{\lambda}(\theta^P)\right] - \mathcal{L}_{\lambda}(\theta_{\lambda}^{*})$
We will assume GLM structure on  $\ell(\cdot)$ , i.e.  $\ell(\theta; z) = f(x^T\theta; y)$ . We will further assume boundedness:  $c \leq f(x^T\theta; y) \leq C$  for some universal constants  $c, C \in \mathbb{R}$ . Applying Taylor's Theorem (in an argument similar to Equation C.8), we can show that for some  $\theta'' \in [\theta^P, \theta_\lambda^*]$
$$
\theta^ {P} - \theta_ {\lambda} ^ {*} = \nabla^ {2} \mathcal {L} _ {\lambda} \left(\theta^ {\prime \prime}\right) ^ {- 1} b.
$$
Note that by the GLM assumption, the eigendecomposition of  $\nabla^2\mathcal{L}_\lambda^* (\theta)$  can be written as  $X^{T}\Lambda (\theta)X$  Then plugging in from Equation H.1 and using the boundedness assumption on the loss  $f$
$$
\begin{array}{l} \mathcal {L} _ {\lambda} \left(\theta^ {P}\right) - \mathcal {L} _ {\lambda} \left(\theta_ {\lambda} ^ {*}\right) = \frac {1}{2} \left| \left| \theta^ {P} - \theta_ {\lambda} ^ {*} \right| \right| _ {\nabla^ {2} \mathcal {L} _ {\lambda} \left(\theta^ {\prime}\right)} ^ {2} \\ = b ^ {T} \left(\nabla^ {2} \mathcal {L} _ {\lambda} \left(\theta^ {\prime \prime}\right)\right) ^ {- 1} \nabla^ {2} \mathcal {L} _ {\lambda} \left(\theta^ {\prime}\right) \left(\nabla^ {2} \mathcal {L} _ {\lambda} \left(\theta^ {\prime \prime}\right)\right) ^ {- 1} b \\ = b ^ {T} \left(X ^ {T} \Lambda \left(\theta^ {\prime \prime}\right) X + \lambda I _ {d}\right) ^ {- 1} \left(X ^ {T} \Lambda \left(\theta^ {\prime \prime}\right) X + \lambda I _ {d}\right) \left(X ^ {T} \Lambda \left(\theta^ {\prime}\right) X + \lambda I _ {d}\right) ^ {- 1} b \\ \leq b ^ {T} c ^ {- 1} \left(X ^ {T} X + \lambda I _ {d}\right) ^ {- 1} C \left(X ^ {T} X + \lambda I _ {d}\right) c ^ {- 1} \left(X ^ {T} X + \lambda I _ {d}\right) ^ {- 1} b \\ \leq \frac {C}{c ^ {2}} \| b \| _ {(X ^ {T} X + \lambda I _ {d}) ^ {- 1}} ^ {2} \\ \end{array}
$$
Then in expectation,
$$
\mathbb {E} \left[ \right. \mathcal {L} _ {\lambda} \left(\theta^ {P}\right) - \mathcal {L} _ {\lambda} \left(\theta_ {\lambda} ^ {*}\right) \leq \frac {C \sigma^ {2}}{c ^ {2}} \operatorname {t r} \left(\left(X ^ {T} X + \lambda I _ {d}\right) ^ {- 1}\right).
$$
I Distance to Optimality
Consider the mechanism  $\mathcal{M}(Z) = f(Z) + \mathcal{N}(0, \sigma^2 I_d)$ , for a function  $f: \mathcal{Z} \to \mathbb{R}^d$  with sensitivity  $\Delta_f = L$ . From Balle & Wang (2018), we know that for any neighboring datasets  $Z$  and  $Z'$ , the privacy loss random variable of this mechanism is distributed as  $\mathcal{N}\left(\frac{\Delta_{Z,Z'}^2}{2\sigma^2}, \frac{\Delta_{Z,Z'}^2}{\sigma^2}\right)$ . Maximizing the Rényi divergence  $D_{\alpha}\left(\mathcal{M}(Z) \parallel \mathcal{M}(Z')\right)$  over all neighboring datasets  $Z \simeq Z'$  shows that the RDP for the Gaussian mechanism can be written as
$$
\begin{array}{l} \epsilon (\alpha) = \frac {1}{\alpha - 1} \log \mathbb {E} \left[ e ^ {(\alpha - 1)} \mathcal {N} \left(\frac {L ^ {2}}{2 \sigma^ {2}}, \frac {L ^ {2}}{\sigma^ {2}}\right) \right] \\ = \frac {L ^ {2}}{2 \sigma^ {2}} + \frac {1}{\alpha - 1} \log \mathbb {E} \left[ e ^ {(\alpha - 1)} \mathcal {N} \left(0, \frac {L ^ {2}}{\sigma^ {2}}\right) \right]. \\ \end{array}
$$
Thus the main deviations between the RDP bound for objective perturbation and that of the Gaussian mechanism are 1) the leading term (a function of  $\beta$  and  $\lambda$  that vanishes as we increase the regularization) and 2) the moment-generating function of the half-normal (instead of normal) distribution.
Figure I plots the Rényi divergence  $\epsilon (\alpha)\coloneqq D_{\alpha}(\mathcal{M}(Z)\parallel \mathcal{M}(Z^{\prime}))$  for the Gaussian mechanism ("normal"), the objective perturbation mechanism ("ObjPert"), and the mechanism obtained by adding noise from the half-normal distribution ("half-normal").