SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\mathbb {E} [ \mathcal {L} (\mathcal {A} (D); D) - \min _ {\mathbf {w} \in \mathcal {C}} \mathcal {L} (\mathbf {w}; D) ] = \widetilde {\Omega} \left(\frac {1}{n ^ {2 / 3}}\right).
$$
For this reason, the utility bound provided above is nearly-optimal.
B.2 Privacy Accounting
Here we prove that our algorithm is  $(\epsilon, \delta)$ -DP. First, note that the sensitivity of each update step is  $\frac{L\lambda}{n}$ , where  $L$  is the Lipschitz constant of the loss function with respect to the  $L_{1}$  norm and  $\lambda$  is the scaling factor of the  $L_{1}$  ball to achieve the constraint region  $\mathcal{C}$ . This is done using Lemma 2.6 from [42], in which we directly bound
$$
\left. \left| \langle s, \nabla \mathcal {L} (\mathbf {w}; D) \rangle - \langle s, \nabla \mathcal {L} (\mathbf {w}; D ^ {\prime}) \rangle \right| \right.
$$
where  $s$  is any vertex of the  $\mathcal{C}$ . Now that we know the sensitivity, we can use the advanced composition theorem for pure differential privacy to find that  $\epsilon = 2\epsilon' \sqrt{2T \log(1/\delta)}$ . Rearranging,  $\epsilon' = \frac{\epsilon}{\sqrt{8T \log(1/\delta)}}$ . Thus, composing  $T$  exponential mechanisms with privacy  $\epsilon'$  produces a final result which is  $(\epsilon, \delta)$ -DP. In our algorithm, we use the Laplace distribution to implement the report noisy maximum version of the exponential mechanism at every iteration [43]. Thus our algorithm is  $(\epsilon, \delta)$ -DP.