SlowGuess/NeurIPS_2023_OCR huggingface dataset:
B.1 Non-adaptive DP optimizers
Proof of Theorem 1. We prove Theorem 1 by showing that, DP-SGD using  $R$ -dependent AUTO-S with learning rate  $\eta$  and weight decay  $\lambda$  is equivalent to  $R$ -independent AUTO-S with learning rate  $\eta R$  and weight decay  $\lambda / R$ . We claim other non-adaptive optimizers such as HeavyBall and NAG can be easily shown in a similar manner.
Recall the standard SGD with weight decay is
$$
\boldsymbol {w} _ {t + 1} = \boldsymbol {w} _ {t} - \eta \left(\sum_ {i \in B _ {t}} \frac {\partial l _ {i}}{\partial \boldsymbol {w} _ {t}} + \lambda \boldsymbol {w} _ {t}\right)
$$
Replacing the standard gradient  $\sum_{i}\frac{\partial l_{i}}{\partial\boldsymbol{w}_{t}}$  with the private gradient, we write the  $R$ -dependent case as
$$
\begin{array}{l} \boldsymbol {w} _ {t + 1} = \boldsymbol {w} _ {t} - \eta \left(\sum_ {i \in B _ {t}} \frac {\partial l _ {i}}{\partial \boldsymbol {w} _ {t}} \cdot R / \| \frac {\partial l _ {i}}{\partial \boldsymbol {w} _ {t}} \| _ {2} + \sigma R \cdot \mathcal {N} (0, \mathbf {I}) + \lambda \boldsymbol {w} _ {t}\right) \\ = \boldsymbol {w} _ {t} - \eta R \left(\sum_ {i \in B _ {t}} \frac {\partial l _ {i}}{\partial \boldsymbol {w} _ {t}} / \| \frac {\partial l _ {i}}{\partial \boldsymbol {w} _ {t}} \| _ {2} + \sigma \cdot \mathcal {N} (0, \mathbf {I})\right) - \eta \lambda \boldsymbol {w} _ {t} \\ \end{array}
$$
which is clearly equivalent to the  $R$ -independent case:
$$
\boldsymbol {w} _ {t + 1} = \boldsymbol {w} _ {t} - \eta^ {\prime} \left(\sum_ {i \in B _ {t}} \frac {\partial l _ {i}}{\partial \boldsymbol {w} _ {t}} / \| \frac {\partial l _ {i}}{\partial \boldsymbol {w} _ {t}} \| _ {2} + \sigma \cdot \mathcal {N} (0, \mathbf {I}) + \lambda^ {\prime} \boldsymbol {w} _ {t}\right)
$$
if we use  $\eta' = \eta R$  and  $\lambda' = \lambda / R$ .
B.2 Adaptive DP optimizers
Proof of Theorem 2. We prove Theorem 2 by showing that, DP-AdamW using  $R$ -dependent AUTO-S with learning rate  $\eta$  and weight decay  $\lambda$  is equivalent to  $R$ -independent AUTO-S with the same learning rate  $\eta$  and weight decay  $\lambda / R$ . This is the most complicated case. We claim other adaptive optimizers such as AdaDelta, Adam with weight decay (not AdamW), and NAdam can be easily shown in a similar manner.
Recall the standard AdamW is
$$
\boldsymbol {w} _ {t + 1} = \boldsymbol {w} _ {t} - \eta \left(\frac {\boldsymbol {m} _ {t} / \left(1 - \beta_ {1}\right)}{\sqrt {\boldsymbol {v} _ {t} / \left(1 - \beta_ {2}\right)}} + \lambda \boldsymbol {w} _ {t}\right)
$$
where  $\beta_{1},\beta_{2}$  are constants,  $\pmb{g}_t\coloneqq \sum_i\frac{\partial l_i}{\partial\pmb{w}_t}$  is the standard gradient,
$$
\mathbf {m} _ {t} = \beta_ {1} \mathbf {m} _ {t - 1} + (1 - \beta_ {1}) \boldsymbol {g} _ {t} \longrightarrow \mathbf {m} _ {t} = \sum_ {\tau} \beta_ {1} ^ {t - \tau} (1 - \beta_ {1}) \boldsymbol {g} _ {\tau},
$$
$$
\boldsymbol {v} _ {t} = \beta_ {2} \boldsymbol {v} _ {t - 1} + (1 - \beta_ {2}) \boldsymbol {g} _ {t} ^ {2} \longrightarrow \boldsymbol {v} _ {t} = \sum_ {\tau} \beta_ {2} ^ {t - \tau} (1 - \beta_ {2}) \boldsymbol {g} _ {\tau} ^ {2}.
$$
Replacing the standard gradient with the private gradient  $R\tilde{g}_t \coloneqq R(\sum_i \frac{\partial l_i}{\partial w_t} / \left\| \frac{\partial l_i}{\partial w_t} \right\|_2 + \sigma \cdot \mathcal{N}(0, I))$ , we write the  $R$ -dependent DP-AdamW as
$$
\boldsymbol {w} _ {t + 1} = \boldsymbol {w} _ {t} - \eta \left(\frac {\tilde {\mathbf {m}} _ {t} / \left(1 - \beta_ {1}\right)}{\sqrt {\tilde {\boldsymbol {v}} _ {t} / \left(1 - \beta_ {2}\right)}} + \lambda \boldsymbol {w} _ {t}\right)
$$
$$
\tilde {\mathbf {m}} _ {t} = \beta_ {1} \tilde {\mathbf {m}} _ {t - 1} + (1 - \beta_ {1}) R \tilde {\boldsymbol {g}} _ {t} \longrightarrow \tilde {\mathbf {m}} _ {t} = \sum_ {\tau} \beta_ {1} ^ {t - \tau} (1 - \beta_ {1}) R \tilde {\boldsymbol {g}} _ {\tau},
$$
$$
\tilde {\boldsymbol {v}} _ {t} = \beta_ {2} \tilde {\boldsymbol {v}} _ {t - 1} + (1 - \beta_ {2}) R ^ {2} \tilde {\boldsymbol {g}} _ {t} ^ {2} \longrightarrow \tilde {\boldsymbol {v}} _ {t} = \sum_ {\tau} \beta_ {2} ^ {t - \tau} (1 - \beta_ {2}) R ^ {2} \tilde {\boldsymbol {g}} _ {\tau} ^ {2}.
$$
Clearly, the  $R$  factor in the numerator and denominator of  $\frac{\tilde{\mathbf{m}}_t / (1 - \beta_1)}{\sqrt{\tilde{\boldsymbol{v}}_t / (1 - \beta_2)}}$  cancel each other. Therefore we claim that the  $R$ -dependent DP-AdamW is in fact completely independent of  $R$ .
B.3 Automatic per-layer clipping
In some cases, the per-layer clipping is desired, where we use a clipping threshold vector  $\mathbf{R} = [R_1, \dots, R_L]$  and each layer uses a different clipping threshold. We claim that DP optimizers under automatic clipping works with the per-layer clipping when  $\mathbf{R}$  is tuned proportionally, e.g.,  $\mathbf{R} = R \cdot [a_1, \dots, a_L]$ , but not entry-wise (see counter-example in Fact B.1). One special case is the uniform per-layer clipping when  $R_1 = \dots = R_L = R / \sqrt{L}$ . This is widely applied as only one norm  $R$  requires tuning, instead of  $L$  norms in  $\mathbf{R}$ , particularly in the case of deep models with hundreds of layers. The corresponding DP-SGD with AUTO-S in (3.5) gives
$$
\boldsymbol {w} _ {t + 1} ^ {(l)} = \boldsymbol {w} _ {t} ^ {(l)} - \eta \left(\sum_ {i \in B _ {t}} \frac {R}{\sqrt {L}} \frac {\boldsymbol {g} _ {t , i} ^ {(l)}}{| | \boldsymbol {g} _ {t , i} ^ {(l)} | | + \gamma} + \sigma R \cdot \mathcal {N} (0, \mathbf {I})\right)
$$
Here the superscript  $(l)$  is the layer index. Clearly  $R$  couples with the learning rate  $\eta$  and the same analysis as in Theorem 1 follows. The adaptive optimizers can be similarly analyzed from Theorem 2.
Fact B.1. Changing one clipping threshold in the clipping threshold vector  $\pmb{R}$  (i.e. not proportionally) can break the coupling with learning rate.
Proof of Fact B.1. We prove by a counter-example of  $\pmb{R}$  in  $\mathbb{R}^2$ . Consider DP-SGD with per-layer clipping thresholds  $(R_1, R_2) = (9, 12)$ :
$$
\boldsymbol {w} _ {t + 1} ^ {(l)} = \boldsymbol {w} _ {t} ^ {(l)} - \eta \left(\sum_ {i \in B} \frac {R _ {l} \boldsymbol {g} _ {t , i , l}}{| | \boldsymbol {g} _ {t , i , l} | |} + \sigma \sqrt {R _ {1} ^ {2} + R _ {2} ^ {2}} \cdot \mathcal {N} (0, \mathbf {I})\right)
$$
Increasing  $R_{1}$  from 9 to 16 changes the update for the first layer
$$
\eta \left(\sum_ {i \in B} \frac {9 \mathbf {g} _ {t , i , l}}{| | \mathbf {g} _ {t , i , l} | |} + 1 5 \sigma \cdot \mathcal {N} (0, 1)\right)\rightarrow \eta \left(\sum_ {i \in B} \frac {1 6 \mathbf {g} _ {t , i , l}}{| | \mathbf {g} _ {t , i , l} | |} + 2 0 \sigma \cdot \mathcal {N} (0, \mathbf {I})\right)
$$
The noise-to-signal ratio decreases from 5/3 to 5/4 for this layer, and increases from 5/4 to 5/3 for the second layer. This breaks the coupling with learning rate, since the coupling does not change the noise-to-signal ratio.
C Main results of convergence for DP-SGD with automatic clipping
C.1 Main proof of convergence for DP-SGD (the envelope version)
Proof of Theorem 4. In this section, we prove two parts of Theorem 4.
The first part of Theorem 4 is the upper bound on  $\min_t\mathbb{E}(\| \pmb {g}_t\|)$ , which is a direct result following from Theorem 6, and we prove it in Appendix C.2.
Theorem 6. Under Assumption 5.1, 5.2, 5.3, running DP-SGD with automatic clipping for  $T$  iterations gives
$$
\min  _ {t} \mathbb {E} \left(\left\| \boldsymbol {g} _ {t} \right\|\right) \leq \frac {\xi}{r} + \mathcal {F} \left(\frac {4}{\sqrt {T}} \sqrt {\left(\mathcal {L} _ {0} - \mathcal {L} _ {*}\right) L \left(1 + \frac {\sigma^ {2} d}{B ^ {2}}\right)}; r, \xi , \gamma\right) \tag {C.1}
$$
- for  $r < 1, \gamma = 0$  and  $\eta \propto 1 / \sqrt{T}$ ,  $\mathcal{F}(x) = \frac{x}{\min_{0 < c < 1} f(c, r)}$  and  $f(c, r) := \frac{(1 + rc)}{\sqrt{r^2 + 2rc + 1}} + \frac{(1 - rc)}{\sqrt{r^2 - 2rc + 1}}$ ; for  $r \geq 1, \gamma = 0$  and  $\eta \propto 1 / \sqrt{T}$ ,  $\mathcal{F}(x) = \infty$ ;
- for  $r \geq 1, \gamma > 0$  and  $\eta \propto 1 / \sqrt{T}$ ,  $\mathcal{F}$  is the convex envelope of (C.9), and is strictly increasing.
Figure 7: Visualization of upper bound  $\frac{\xi}{r} +\mathcal{F}\left(O(1 / \sqrt{T});r,\xi ,\gamma\right)$  for gradient norm, with  $O(1 / \sqrt{T})$  in (C.1). Here  $\xi = 1$ . The right plot is a zoom-in (with additional lines) of the left one.
Notice that, (C.1) holds for any  $r > 0$ . However, we have to consider an envelope curve over  $r$  in (C.1) to reduce the upper bound: with AUTO-V clipping ( $\gamma = 0$ ), the upper bound in (C.1) is always larger than  $\xi$  as  $r < 1$ ; we must use AUTO-S clipping ( $\gamma > 0$ ) to reduce the upper bound to zero, as can be seen from Figure 7. In fact, larger  $T$  needs larger  $r$  to reduce the upper bound.
All in all, we specifically focus on  $r \geq 1$  and  $\gamma > 0$ , which is the only scenario that (C.1) can converge to zero. This scenario is also where we prove the second part of Theorem 4.
The second part of Theorem 4 is the asymptotic convergence rate  $O(T^{-1/4})$  of DP-SGD, only possible under  $r \geq 1$  and  $\gamma > 0$ .
By (C.1) in Theorem 6, our upper bound  $\mathcal{G}$  from Theorem 4 can be simplified to
$$
\min  _ {r > 0} \frac {\xi}{r} + (\mathcal {M} ^ {- 1}) _ {c c v} \left(\frac {4}{\sqrt {T}} \sqrt {(\mathcal {L} _ {0} - \mathcal {L} _ {*}) L \left(1 + \frac {\sigma^ {2} d}{B ^ {2}}\right)}; r, \xi , \gamma\right)
$$
where the function  $\mathcal{M}^{-1}$  is explicitly defined in (C.9) and the subscript  $ccv$  means the upper concave envelope. Clearly, as  $T\to \infty$ ,  $\mathcal{M}^{-1}(\frac{1}{\sqrt{T}})\to 0$ . We will next show that the convergence rate of  $\mathcal{M}^{-1}$  is indeed  $O(\frac{1}{\sqrt{T}})$  and the minimization over  $r$  makes the overall convergence rate  $O(T^{-1 / 4})$ .
Starting from (C.9), we denote  $x = \frac{4}{\sqrt{T}} \sqrt{\left(\mathcal{L}_0 - \mathcal{L}_*\right) L\left(1 + \frac{\sigma^2 d}{B^2}\right)}$  and write
$$
\begin{array}{l} \mathcal {M} ^ {- 1} (x; r, \xi , \gamma) = \frac {- \frac {\xi}{r} \gamma + (r ^ {2} - 1) \frac {\xi}{r} x + r \gamma x + \gamma \sqrt {(\frac {\xi}{r}) ^ {2} + 2 \xi x + 2 \gamma x + x ^ {2}}}{2 \gamma - (r ^ {2} - 1) x} \\ = \left(- \frac {\gamma \xi}{r} + (r ^ {2} - 1) \frac {\xi}{r} x + r \gamma x + \gamma \sqrt {\left(\frac {\xi}{r}\right) ^ {2} + 2 \xi x + 2 \gamma x + x ^ {2}}\right) \\ \cdot \frac {1 + \frac {r ^ {2} - 1}{2 \gamma} x + O (x ^ {2})}{2 \gamma} \\ = \frac {1}{2 \gamma} \left(- \frac {\gamma \xi}{r} + (r ^ {2} - 1) \frac {\xi}{r} x + r \gamma x + \frac {\gamma \xi}{r} \sqrt {1 + \frac {2 (\xi + \gamma) r ^ {2} x}{\xi^ {2}} + O (x ^ {2})}\right) \\ \cdot \left(1 + \frac {r ^ {2} - 1}{2 \gamma} x + O \left(x ^ {2}\right)\right) \\ = \frac {1}{2 \gamma} \left(- \frac {\gamma \xi}{r} + (r ^ {2} - 1) \frac {\xi}{r} x + r \gamma x + \frac {\gamma \xi}{r} \left(1 + \frac {(\xi + \gamma) r ^ {2} x}{\xi^ {2}} + O \left(x ^ {2}\right)\right)\right) \\ \cdot \left(1 + \frac {r ^ {2} - 1}{2 \gamma} x + O \left(x ^ {2}\right)\right) \\ = \frac {1}{2 \gamma} \left((r ^ {2} - 1) \frac {\xi}{r} x + r \gamma x + \frac {\gamma (\xi + \gamma) r x}{\xi} + O (x ^ {2})\right) \cdot \left(1 + \frac {r ^ {2} - 1}{2 \gamma} x + O (x ^ {2})\right) \\ = \frac {1}{2 \gamma} \left((r ^ {2} - 1) \frac {\xi}{r} + r \gamma + \frac {\gamma (\xi + \gamma) r}{\xi}\right) \cdot x + O (x ^ {2}) \\ = \frac {1}{2 \gamma} \left(\frac {(\xi + \gamma) ^ {2}}{\xi} r - \frac {\xi}{r}\right) \cdot x + O (x ^ {2}) \\ \end{array}
$$
Since  $\mathcal{M}^{-1}$  is asymptotically linear as  $x\to 0$  , we instead study
$$
\min _ {r > 0} \frac {\xi}{r} + \mathcal {M} ^ {- 1} (x; r, \xi , \gamma) \equiv \min _ {r > 0} \frac {\xi}{r} + \frac {1}{2 \gamma} \left(\frac {(\xi + \gamma) ^ {2}}{\xi} r - \frac {\xi}{r}\right) \cdot x + O (x ^ {2}).
$$
That is, ignoring the higher order term for the asymptotic analysis, the  $\mathcal{M}^{-1}$  part converges as  $O(x) = O(1 / \sqrt{T})$ , and we visualize this in Figure 9.
Although DP-SGD converges faster than SGD, the former converges to  $\xi /r$  and the latter converges to 0. Thus, taking  $\xi /r$  into consideration, the objective reduces to a hyperbola
$$
\frac {\left(\xi \left(1 - \frac {x}{2 \gamma}\right)\right)}{r} + \frac {x (\xi + \gamma) ^ {2}}{2 \gamma \xi} \cdot r
$$
whose minimum over  $r$  is obviously  $2\sqrt{\xi(1 - \frac{x}{2\gamma})\frac{x(\xi + \gamma)^2}{2\gamma\xi}} = O(\sqrt{x}) = O(T^{-1/4})$ .
To give more details about the upper bound in (5.2), we demonstrate its dependence on  $\xi$  and  $\gamma$  in Figure 8.
C.2 Main proof of convergence for DP-SGD (the non-envelope version)
Proof of Theorem 6. Consider DP-SGD with AUTO-S clipping
$$
\boldsymbol {w} _ {t + 1} = \boldsymbol {w} _ {t} - \eta \left(\sum_ {i} \frac {\tilde {\boldsymbol {g}} _ {t , i}}{\| \tilde {\boldsymbol {g}} _ {t , i} \| + \gamma} + \sigma \mathcal {N} (0, \mathbf {I})\right)
$$
where  $\tilde{g}_{t,i}$  is i.i.d. samples of  $\tilde{g}_t$ , an unbiased estimate of  $g_{t}$ , with a bounded variance as described in Assumption 5.3.
Figure 8: Dependence of the upper bound  $\mathcal{G}$  on  $\xi$  (left) and  $\gamma$  (right). Here the  $O(1 / \sqrt{T})$  term is set to 10 and either  $\gamma = 0.01$  (left) or  $\xi = 1$  (right).
Figure 9: Convergence with respect to  $T$ . Same setting as Figure 6.
By Lipschitz smoothness in Assumption 5.2, and denoting  $Z = \mathcal{N}(0,\mathbf{I})$ , we have
$$
\begin{array}{l} \mathcal {L} _ {t + 1} - \mathcal {L} _ {t} \leq \boldsymbol {g} _ {t} ^ {\top} (\boldsymbol {w} _ {t + 1} - \boldsymbol {w} _ {t}) + \frac {L}{2} \| \boldsymbol {w} _ {t + 1} - \boldsymbol {w} _ {t} \| ^ {2} \\ = - \eta \boldsymbol {g} _ {t} ^ {\top} \left(\sum_ {i} C _ {i} \tilde {\boldsymbol {g}} _ {t, i} + \sigma Z\right) + \frac {L \eta^ {2}}{2} \left\| \sum_ {i} C _ {i} \tilde {\boldsymbol {g}} _ {t, i} + \sigma Z \right\| ^ {2} \\ \leq - \eta \boldsymbol {g} _ {t} ^ {\top} \left(\sum_ {i} C _ {i} \tilde {\boldsymbol {g}} _ {t, i} + \sigma Z\right) + L \eta^ {2} \left(\left\| \sum_ {i} C _ {i} \tilde {\boldsymbol {g}} _ {t, i} \right\| ^ {2} + \sigma^ {2} \| Z \| ^ {2}\right) \tag {C.2} \\ \leq - \eta \boldsymbol {g} _ {t} ^ {\top} \left(\sum_ {i} C _ {i} \tilde {\boldsymbol {g}} _ {t, i} + \sigma Z\right) + L \eta^ {2} \left(B ^ {2} + \sigma^ {2} \| Z \| ^ {2}\right) \\ \end{array}
$$
where the second last inequality follows from Cauchy Schwartz, and the last inequality follows from the fact that  $\| C_i\tilde{g}_{t,i}\| \leq 1$ , e.g.  $C_i$  is  $\| \tilde{g}_{t,i} / (\| \tilde{g}_{t,i}\| +\gamma)\|$  or the re-parameterized clipping in [17].
Notice that in the last equality, the first term (ignoring  $g_t^\top Z$  for its zero expectation) can be written in the same form as (3.3), which supports our motivation in Section 3.2; the second term is independent of clipping functions. Note that the last inequality is tight if and only if  $C_i = 1$ . This empirically holds in Appendix H.1, especially for GPT2.
Given the fact that  $\| \tilde{g}_{t,i} / (\| \tilde{g}_{t,i}\| +\gamma)\| \leq 1$  , the expected improvement at one iteration is
$$
\begin{array}{l} \mathbb {E} \left(\mathcal {L} _ {t + 1} - \mathcal {L} _ {t} | \boldsymbol {w} _ {t}\right) \leq - \eta \boldsymbol {g} _ {t} ^ {\top} \mathbb {E} \left(\sum_ {i} \frac {\tilde {\boldsymbol {g}} _ {t , i}}{\| \tilde {\boldsymbol {g}} _ {t , i} \| + \gamma}\right) + L \eta^ {2} (B ^ {2} + \sigma^ {2} d) \tag {C.3} \\ = - \eta B \boldsymbol {g} _ {t} ^ {\top} \mathbb {E} \left(\frac {\tilde {\boldsymbol {g}} _ {t}}{\| \tilde {\boldsymbol {g}} _ {t} \| + \gamma}\right) + L \eta^ {2} (B ^ {2} + \sigma^ {2} d) \\ \end{array}
$$
Now we want to lower bound  $\pmb{g}_t^\top \mathbb{E}\left(\frac{\tilde{\pmb{g}}_t}{\|\tilde{\pmb{g}}_t\| + \gamma}\right)$  in (C.3).
Write  $\tilde{\pmb{g}}_t = \pmb {g}_t + \Delta_t$  where the gradient noise  $\Delta_{t}$  follows  $\mathbb{E}\Delta_t = 0,\mathbb{E}\| \Delta_t\| <  \xi$  by Assumption 5.3. Then
$$
\begin{array}{l} \boldsymbol {g} _ {t} ^ {\top} \mathbb {E} \left(\frac {\tilde {\boldsymbol {g}} _ {t}}{\| \tilde {\boldsymbol {g}} _ {t} \| + \gamma}\right) = \mathbb {E} \left(\frac {\| \boldsymbol {g} _ {t} \| ^ {2} + \boldsymbol {g} _ {t} ^ {\top} \Delta_ {t}}{\| \boldsymbol {g} _ {t} + \Delta_ {t} \| + \gamma}\right) \\ = \frac {1}{2} \mathbb {E} \left(\frac {\| \boldsymbol {g} _ {t} \| ^ {2} + \boldsymbol {g} _ {t} ^ {\top} \Delta_ {t}}{\| \boldsymbol {g} _ {t} + \Delta_ {t} \| + \gamma} \Big | \Delta_ {t} \in H _ {+}\right) + \frac {1}{2} \mathbb {E} \left(\frac {\| \boldsymbol {g} _ {t} \| ^ {2} + \boldsymbol {g} _ {t} ^ {\top} \Delta_ {t}}{\| \boldsymbol {g} _ {t} + \Delta_ {t} \| + \gamma} \Big | \Delta_ {t} \in H _ {-}\right) \\ = \frac {1}{2} \mathbb {E} \left(\frac {\| \boldsymbol {g} _ {t} \| ^ {2} + \boldsymbol {g} _ {t} ^ {\top} \Delta_ {t}}{\| \boldsymbol {g} _ {t} + \Delta_ {t} \| + \gamma} \Big | \Delta_ {t} \in H _ {+}\right) + \frac {1}{2} \mathbb {E} \left(\frac {\| \boldsymbol {g} _ {t} \| ^ {2} - \boldsymbol {g} _ {t} ^ {\top} \Delta_ {t}}{\| \boldsymbol {g} _ {t} - \Delta_ {t} \| + \gamma} \Big | \Delta_ {t} \in H _ {+}\right) \\ \end{array}
$$
where we use the hyperplane perpendicular to  $g_{t}$  to divide the support of  $\Delta_t$  into two half-spaces:
$$
H _ {+} := \left\{\boldsymbol {v}: \boldsymbol {g} _ {t} ^ {\top} \boldsymbol {v} > 0 \right\}, \quad H _ {-} := \left\{\boldsymbol {v}: \boldsymbol {g} _ {t} ^ {\top} \boldsymbol {v} <   0 \right\}.
$$
We use the symmetry assumption in Assumption 5.3 to get
$$
\mathbb {P} (\Delta_ {t} \in H _ {+}) = \mathbb {P} (\Delta_ {t} \in H _ {-}) = \frac {1}{2}
$$
and notice that  $\Delta_t \stackrel{D}{=} -\Delta_t$ , i.e., if  $\Delta_t \in H_+$ , then  $-\Delta_t \in H_-$  with the same distribution.
The next result further gives a lower bound for  $\pmb{g}_t^\top \mathbb{E}\left(\frac{\tilde{\pmb{g}}_t}{\|\tilde{\pmb{g}}_t\| + \gamma}\right)$  using  $\| \pmb {g}_t\|$ .
Lemma C.1.
$$
\mathbb {E} \left(\frac {\| \boldsymbol {g} _ {t} \| ^ {2} + \boldsymbol {g} _ {t} ^ {\top} \Delta_ {t}}{\| \boldsymbol {g} _ {t} + \Delta_ {t} \| + \gamma} + \frac {\| \boldsymbol {g} _ {t} \| ^ {2} - \boldsymbol {g} _ {t} ^ {\top} \Delta_ {t}}{\| \boldsymbol {g} _ {t} - \Delta_ {t} \| + \gamma} | \Delta_ {t} \in H _ {+}\right) \geq \min  _ {0 <   c \leq 1} f (c, r; \frac {\gamma}{\| \boldsymbol {g} _ {t} \|}) \cdot (\| \boldsymbol {g} _ {t} \| - \xi / r)
$$
for any  $r > 0$  and  $f(c, r; \Gamma) = \frac{(1 + rc)}{\sqrt{r^2 + 2rc + 1} + \Gamma} + \frac{(1 - rc)}{\sqrt{r^2 - 2rc + 1} + \Gamma}$ .
For the simplicity of notation, we denote the distance measure
$$
\mathcal {M} \left(\left\| \boldsymbol {g} _ {t} \right\| - \xi / r; r, \xi , \gamma\right) = \min  _ {0 <   c \leq 1} f \left(c, r; \frac {\gamma}{\left\| \boldsymbol {g} _ {t} \right\|}\right) \cdot \left(\left\| \boldsymbol {g} _ {t} \right\| - \xi / r\right) \tag {C.4}
$$
and leave the fine-grained analysis (e.g. its explicit form in some scenarios) at the end of this section.
Using the lower bound from Lemma C.1, the expected improvement (C.3) becomes
$$
\mathbb {E} \left(\mathcal {L} _ {t + 1} - \mathcal {L} _ {t} | \boldsymbol {w} _ {t}\right) \leq - \frac {\eta B}{2} \mathcal {M} \left(\left\| \boldsymbol {g} _ {t} \right\| - \xi / r\right) + L \eta^ {2} B ^ {2} \left(1 + \frac {\sigma^ {2} d}{B ^ {2}}\right)
$$
Now extend the expectation over randomness in the trajectory, and perform a telescoping sum over the iterations
$$
\begin{array}{l} \mathcal {L} _ {0} - \mathcal {L} _ {*} \geq \mathcal {L} _ {0} - \mathbb {E} \mathcal {L} _ {T} = \sum_ {t} \mathbb {E} \left(\mathcal {L} _ {t} - \mathcal {L} _ {t + 1}\right) \\ \geq \frac {\eta B}{2} \mathbb {E} \left(\sum_ {t} \mathcal {M} (\| \boldsymbol {g} _ {t} \| - \xi / r)\right) - T L \eta^ {2} B ^ {2} \left(1 + \frac {\sigma^ {2} d}{B ^ {2}}\right) \\ \end{array}
$$
Substituting  $\eta B = \eta_0 / \sqrt{T}$  where  $\eta_0$  is a base learning rate, we have
$$
2 \left(\mathcal {L} _ {0} - \mathcal {L} _ {*}\right) \geq \sqrt {T} \eta_ {0} \mathbb {E} \left(\frac {1}{T} \sum_ {t} \mathcal {M} \left(\| \boldsymbol {g} _ {t} \| - \xi / r\right)\right) - 2 L \eta_ {0} ^ {2} \left(1 + \frac {\sigma^ {2} d}{B ^ {2}}\right)
$$
$$
\mathbb {E} \left(\frac {1}{T} \sum_ {t} \mathcal {M} \left(\left\| \boldsymbol {g} _ {t} \right\| - \xi / r\right)\right) \leq \frac {1}{\sqrt {T}} \left[ \frac {2 \left(\mathcal {L} _ {0} - \mathcal {L} _ {*}\right)}{\eta_ {0}} + 2 L \eta_ {0} \left(1 + \frac {\sigma^ {2} d}{B ^ {2}}\right) \right] \tag {C.5}
$$
With  $\eta_0$  chosen properly at  $\eta_0 = \sqrt{\frac{\mathcal{L}_0 - \mathcal{L}_*}{L\left(1 + \frac{\sigma^2d}{B^2}\right)}$ , the hyperbola on the right hand side in (C.5) is minimized to  $4\sqrt{(\mathcal{L}_0 - \mathcal{L}_*)L\left(1 + \frac{\sigma^2d}{B^2}\right)}$ , and we obtain
$$
\mathbb {E} \left(\frac {1}{T} \sum_ {t} \mathcal {M} (\| \boldsymbol {g} _ {t} \| - \xi / r)\right) \leq \frac {4}{\sqrt {T}} \sqrt {\left(\mathcal {L} _ {0} - \mathcal {L} _ {*}\right) L \left(1 + \frac {\sigma^ {2} d}{B ^ {2}}\right)}
$$
Since the minimum of a sequence is smaller than the average, we have
$$
\min  _ {t} \mathbb {E} (\mathcal {M} (\| \boldsymbol {g} _ {t} \| - \xi / r)) \leq \frac {1}{T} \sum_ {t} \mathbb {E} \left(\mathcal {M} (\| \boldsymbol {g} _ {t} \| - \xi / r)\right) \leq \frac {4}{\sqrt {T}} \sqrt {\left(\mathcal {L} _ {0} - \mathcal {L} _ {*}\right) L \left(1 + \frac {\sigma^ {2} d}{B ^ {2}}\right)} \tag {C.6}
$$
We claim that  $\mathcal{M}$  may not be concave or convex. Therefore we use  $\mathcal{M}_{cvx}$  to denote its lower convex envelope, i.e. the largest convex function that is smaller than  $\mathcal{M}$ . Then by Jensen's inequality (C.6) becomes
$$
\min  _ {t} \mathcal {M} _ {c v x} (\mathbb {E} (\| \boldsymbol {g} _ {t} \| - \xi / r)) \leq \min  _ {t} \mathbb {E} (\mathcal {M} _ {c v x} (\| \boldsymbol {g} _ {t} \| - \xi / r)) \leq \frac {4}{\sqrt {T}} \sqrt {\left(\mathcal {L} _ {0} - \mathcal {L} _ {*}\right) L \left(1 + \frac {\sigma^ {2} d}{B ^ {2}}\right)} \tag {C.7}
$$
It is obvious that  $\mathcal{M}_{cvx}$  is increasing as  $\mathcal{M}$  is increasing by Theorem 8. Hence,  $(\mathcal{M}_{cvx})^{-1}$  is also increasing, as the inverse of  $\mathcal{M}_{cvx}$ . We write (C.7) as
$$
\min  _ {t} \mathbb {E} \left(\| \boldsymbol {g} _ {t} \| - \xi / r\right) \leq \left(\mathcal {M} _ {c v x}\right) ^ {- 1} \left(\frac {4}{\sqrt {T}} \sqrt {\left(\mathcal {L} _ {0} - \mathcal {L} _ {*}\right) L \left(1 + \frac {\sigma^ {2} d}{B ^ {2}}\right)}\right)
$$
and equivalently
$$
\min  _ {t} \mathbb {E} \left(\left\| \boldsymbol {g} _ {t} \right\|\right) \leq \frac {\xi}{r} + \left(\mathcal {M} _ {c v x}\right) ^ {- 1} \left(\frac {4}{\sqrt {T}} \sqrt {\left(\mathcal {L} _ {0} - \mathcal {L} _ {*}\right) L \left(1 + \frac {\sigma^ {2} d}{B ^ {2}}\right)}\right) \tag {C.8}
$$
Finally, we derive the explicit properties of  $\mathcal{M}(\| \pmb{g}_t\| -\xi /r)$  in Theorem 8. These properties allow us to further analyze on the convergence of  $\mathcal{M}(\| \pmb {g}_t\| -\xi /r)$ , based on AUTO-V and AUTO-S, respectively.
1. DP-SGD with AUTO-V clipping. By Theorem 8, we write
$$
\mathcal {M} (x; r) = \min  _ {c \in (0, 1 ]} f (c, r; 0) \cdot x
$$
This is a linear function and thus  $\mathcal{M}_{cvx} = \mathcal{M} = 1 / \mathcal{M}_{cvx}^{-1}$ . As a result, we have
$$
\min  _ {t} \mathbb {E} (\| \boldsymbol {g} _ {t} \|) \leq \frac {\xi}{r} + \frac {1}{\min  _ {c \in (0 , 1 ]} f (c , r ; 0)} \cdot \frac {4}{\sqrt {T}} \sqrt {\left(\mathcal {L} _ {0} - \mathcal {L} _ {*}\right) L \left(1 + \frac {\sigma^ {2} d}{B ^ {2}}\right)}
$$
We note here  $r$  plays an important role under AUTO-V clipping: when  $r < 1$ , we spend more iterations to converge to better and smaller gradient norm  $\xi / r$ ; when  $r \geq 1$ ,  $\min_c f(c, r; 0) = f(1, r; 0) = 0$  and it takes forever to converge. This is demonstrated in the left plot of Figure 6.
2. DP-SGD with AUTO-S clipping. By Theorem 8 and for  $r > 1$ , we write
$$
\mathcal {M} (x; r, \xi , \gamma) = \left(\frac {\gamma}{(r - 1) (x + \xi / r) + \gamma} - \frac {\gamma}{(r + 1) (x + \xi / r) + \gamma}\right) \cdot x.
$$