SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Notice that the inverse of a lower convex envelope is equivalent to the upper concave envelope (denoted by the subscript  $ccv$ ) of an inverse. Therefore we can derive  $(\mathcal{M}_{cvx})^{-1} = (\mathcal{M}^{-1})_{cv}$  with the explicit form
$$
\mathcal {M} ^ {- 1} (x; r, \xi , \gamma) = \frac {- \frac {\xi}{r} \gamma + (r ^ {2} - 1) \frac {\xi}{r} x + r \gamma x + \gamma \sqrt {(\frac {\xi}{r}) ^ {2} + 2 \xi x + 2 \gamma x + x ^ {2}}}{2 \gamma - (r ^ {2} - 1) x}. \tag {C.9}
$$
we can derive it based on  $r,\xi ,\gamma$  and substitute back to (C.8).
Note that the domain of  $\mathcal{M}^{-1}$  (or the image of  $\mathcal{M}$ ) is  $[0, \frac{\gamma}{r - 1} - \frac{\gamma}{r + 1})$ .
In comparison to the AUTO-V clipping,  $\mathcal{M}^{-1}$  takes a much more complicated form, as depicted in the middle plot of Figure 6, where  $r > 1$  plays an important role for the gradient norm to converge to zero.
C.3 Proof of Lemma C.1
Proof of Lemma C.1. We want to lower bound
$$
\mathbb {E} \left(\frac {\left\| \boldsymbol {g} _ {t} \right\| ^ {2} + \boldsymbol {g} _ {t} ^ {\top} \Delta_ {t}}{\left\| \boldsymbol {g} _ {t} + \Delta_ {t} \right\| + \gamma} + \frac {\left\| \boldsymbol {g} _ {t} \right\| ^ {2} - \boldsymbol {g} _ {t} ^ {\top} \Delta_ {t}}{\left\| \boldsymbol {g} _ {t} - \Delta_ {t} \right\| + \gamma} \mid \Delta_ {t} \in H _ {+}\right) \tag {C.10}
$$
To simplify the notation, we denote noise-to-signal ratio  $S \coloneqq \frac{\|\Delta_t\|}{\|\pmb{g}_t\|}$  and  $c \coloneqq \cos \theta = \frac{\pmb{g}_t^\top\Delta_t}{\|\pmb{g}_t\| \|\Delta_t\|}$ , with  $\theta$  be the random angle between  $\pmb{g}_t$  and  $\Delta_{t}$ . Note that  $0 < c \leq 1$  when  $\Delta_t \in H_+$ .
The term inside the conditional expectation in (C.10) can be written as
$$
\begin{array}{l} \frac {(1 + S c) \| \boldsymbol {g} _ {t} \| ^ {2}}{\sqrt {S ^ {2} + 2 S c + 1} \| \boldsymbol {g} _ {t} \| + \gamma} + \frac {(1 - S c) \| \boldsymbol {g} _ {t} \| ^ {2}}{\sqrt {S ^ {2} - 2 S c + 1} \| \boldsymbol {g} _ {t} \| + \gamma} \\ = \| \boldsymbol {g} _ {t} \| \left(\frac {(1 + S c)}{\sqrt {S ^ {2} + 2 S c + 1} + \gamma / \| \boldsymbol {g} _ {t} \|} + \frac {(1 - S c)}{\sqrt {S ^ {2} - 2 S c + 1} + \gamma / \| \boldsymbol {g} _ {t} \|}\right) \\ \end{array}
$$
Defining  $\Gamma = \gamma / \| \pmb{g}_t\|$  and
$$
f (c, S; \Gamma) := \frac {(1 + S c)}{\sqrt {S ^ {2} + 2 S c + 1} + \Gamma} + \frac {(1 - S c)}{\sqrt {S ^ {2} - 2 S c + 1} + \Gamma}, \tag {C.11}
$$
we turn the conditional expectation in (C.10) into
$$
\mathbb {E} \left(\frac {\left\| \boldsymbol {g} _ {t} \right\| ^ {2} + \boldsymbol {g} _ {t} ^ {\top} \Delta_ {t}}{\left\| \boldsymbol {g} _ {t} + \Delta_ {t} \right\| + \gamma} + \frac {\left\| \boldsymbol {g} _ {t} \right\| ^ {2} - \boldsymbol {g} _ {t} ^ {\top} \Delta_ {t}}{\left\| \boldsymbol {g} _ {t} - \Delta_ {t} \right\| + \gamma} \mid \Delta_ {t} \in H _ {+}\right) = \| \boldsymbol {g} _ {t} \| \mathbb {E} (f (c, S; \Gamma) | \Delta_ {t} \in H _ {+}) \tag {C.12}
$$
for which we want to lower bound  $f(c, S; \Gamma)$  over  $0 < c \leq 1$ ,  $S > 0$ ,  $\Gamma > 0$ . We use the next theorem to prepare some helpful properties. The proof can be found in Appendix E.1.
Theorem 7. For  $f$  defined in (C.11), we have
1.  $f(c, S; \Gamma)$  is strictly decreasing in  $S$  for all  $0 < c < 1$  and  $\Gamma > 0$ .
2. Consequently,  $\min_{c\in (0,1)}f(c,S;\Gamma)$  is strictly decreasing in  $S$
3.  $f(c, S; \Gamma)$  is strictly decreasing in  $c$  for all  $S > 1$  and  $\Gamma > 0$ .
We consider a thresholding ratio  $r > 0$  and we will focus on the regime that  $S < r$ . This  $r$  will turn out to measure the minimum gradient norm at convergence: informally speaking,  $\| \pmb{g}_t\|$  converges to  $\xi /r$ .
By the law of total expectation, (C.12) can be relaxed as follows.
$$
\begin{array}{l} \| \boldsymbol {g} _ {t} \| \mathbb {E} \left(f (c, S; \Gamma) | \Delta \in H _ {+}\right) \\ = \| \boldsymbol {g} _ {t} \| \mathbb {E} \left(f (c, S; \Gamma) | \Delta \in H _ {+}, S <   r\right) \mathbb {P} (r \| \boldsymbol {g} _ {t} \| > \| \Delta \| | \Delta \in H _ {+}) \\ + \| \boldsymbol {g} _ {t} \| \mathbb {E} \left(f (c, S; \Gamma) | \Delta \in H _ {+}, S > r\right) \mathbb {P} (r \| \boldsymbol {g} _ {t} \| <   \| \Delta \| | \Delta \in H _ {+}) \\ \geq \| \boldsymbol {g} _ {t} \| \mathbb {E} \left(f (c, S; \Gamma) | \Delta \in H _ {+}, S <   r\right) \mathbb {P} (r \| \boldsymbol {g} _ {t} \| > \| \Delta \| | \Delta \in H _ {+}) \tag {C.13} \\ \geq \| \boldsymbol {g} _ {t} \| \mathbb {E} \left(f (c, r; \Gamma) | \Delta \in H _ {+}, S <   r\right) \mathbb {P} (r \| \boldsymbol {g} _ {t} \| > \| \Delta \| | \Delta \in H _ {+}) \\ = \| \boldsymbol {g} _ {t} \| \mathbb {E} \left(f (c, r; \Gamma) | \Delta \in H _ {+}, S <   r\right) \mathbb {P} (r \| \boldsymbol {g} _ {t} \| > \| \Delta \|) \\ \geq \min  _ {c \in (0, 1 ]} f (c, r; \Gamma) \cdot \underbrace {\| \boldsymbol {g} _ {t} \| \mathbb {P} (r \| \boldsymbol {g} _ {t} \| > \| \Delta \|)} _ {\star} \\ \end{array}
$$
where in the first inequality, the ignoring of last term is justified by  $f(c, S; \Gamma) \geq \min_{c \in (0,1]} f(c, S; \Gamma) \geq \min_{c \in (0,1]} f(c, \infty; \Gamma) = 0$ , from the monotonicity (second statement) in Theorem 7.
We first lower bound  $\hat{\star}$  by applying the Markov's inequality:
$$
\mathbb {P} (r \| \boldsymbol {g} _ {t} \| > \| \Delta_ {t} \|) \geq 1 - \frac {\mathbb {E} \| \Delta_ {t} \|}{r \| \boldsymbol {g} _ {t} \|}
$$
and hence by Assumption 5.3,
$$
\left\| \boldsymbol {g} _ {t} \right\| \mathbb {P} \left(r \left\| \boldsymbol {g} _ {t} \right\| > \left\| \Delta_ {t} \right\|\right) \geq \left\| \boldsymbol {g} _ {t} \right\| - \mathbb {E} \| \Delta \| / r \geq \left\| \boldsymbol {g} _ {t} \right\| - \xi / r.
$$
Finally, the conditional expectation of interest in (C.10) gives
$$
\mathbb {E} \left(\frac {\| \boldsymbol {g} _ {t} \| ^ {2} + \boldsymbol {g} _ {t} ^ {\top} \Delta_ {t}}{\| \boldsymbol {g} _ {t} + \Delta_ {t} \|} + \frac {\| \boldsymbol {g} _ {t} \| ^ {2} - \boldsymbol {g} _ {t} ^ {\top} \Delta_ {t}}{\| \boldsymbol {g} _ {t} - \Delta_ {t} \|} \Big | \Delta_ {t} \in H _ {+}\right) \geq \min  _ {0 <   c \leq 1} f (c, r; \frac {\gamma}{\| \boldsymbol {g} _ {t} \|}) \cdot (\| \boldsymbol {g} _ {t} \| - \xi / r)
$$
C.4 Proof of Theorem 8
To derive some properties of  $\min_c f(c, r; \Gamma)$ , we need to compute separately for AUTO-V (without the stability constant,  $\Gamma = 0$ ) and for AUTO-S (with the stability constant,  $\Gamma > 0$ ), as shown in Theorem 8. As we will show, as the number of training iterations  $T \to \infty$ , DP-SGD with AUTO-V clipping can only compress  $\| g_t\|$  to  $\xi / r$  for  $r < 1$ . However, DP-SGD with AUTO-S clipping can compress  $\| g_t\|$  to  $\xi / r$  to any  $r > 1$ .
Theorem 8.
1. For  $0 < r < 1$  and  $\Gamma = 0$ , we have  $\min_{c \in (0,1]} f(c, r; 0) > 0$ . Then Equation (C.12) is lower bounded by
$$
\min  _ {c \in (0, 1 ]} f (c, r; 0) \cdot (\| \boldsymbol {g} _ {t} \| - \xi / r)
$$
which is increasing in  $\| g\| -\xi /r$
2. For  $r \geq 1$  and  $\Gamma = 0$ , we have  $\min_{c \in (0,1]} f(c, r; \Gamma) = f(1, r; 0) = 0$ . In words, (C.10) has a trivial lower bound and Theorem 6 cannot compress  $\| \pmb{g}_t \|$  to  $\xi / r$ .
3. For  $r \geq 1$  and  $\Gamma > 0$ , we have  $\min_{c \in (0,1]} f(c, r; \Gamma) = f(1, r; \Gamma) = \left( \frac{\Gamma}{r + \Gamma - 1} - \frac{\Gamma}{r + \Gamma + 1} \right)$ . Then Equation (C.12) is lower bounded by
$$
\left(\frac {\gamma}{(r - 1) \left\| \boldsymbol {g} _ {t} \right\| + \gamma} - \frac {\gamma}{(r + 1) \left\| \boldsymbol {g} _ {t} \right\| + \gamma}\right) \cdot \left(\left\| \boldsymbol {g} _ {t} \right\| - \xi / r\right)
$$
which is increasing in  $\| \pmb{g}_t\| -\xi /r$
Proof. To prove statement 1, we use the second statement from Theorem 7 and show that  $\min_c f(c,r;0) > \min_c f(c,\infty ;0) = 0$ . To prove statement 2 and 3, we use the third statement from Theorem 7 and see that  $\min_c f(c,r;\Gamma) = f(1,r;\Gamma)$  with an explicit formula.
D Convergence rate of standard SGD
Theorem 9. Under Assumption 5.1, 5.2, 5.3 (without the symmetry assumption), running the standard non-DP SGD for  $T$  iterations gives, for  $\eta \propto 1 / \sqrt{T}$
$$
\min _ {t} \mathbb {E} \left(\| \pmb {g} _ {t} \|\right) \leq \frac {1}{T ^ {1 / 4}} \sqrt {2 (\mathcal {L} _ {0} - \mathcal {L} _ {*}) L + \frac {\xi^ {2}}{B}}
$$
Proof of Theorem 9. Consider the standard SGD
$$
\boldsymbol {w} _ {t + 1} = \boldsymbol {w} _ {t} - \eta \frac {\sum_ {i} \tilde {\boldsymbol {g}} _ {t , i}}{B}
$$
where  $\tilde{g}_{t,i}$  is i.i.d. unbiased estimate of  $\pmb{g}_t$ , with a bounded variance as described in Assumption 5.3.
By Lipschitz smoothness assumption in Assumption 5.2,
$$
\mathcal {L} _ {t + 1} - \mathcal {L} _ {t} \leq \boldsymbol {g} _ {t} ^ {\top} (\boldsymbol {w} _ {t + 1} - \boldsymbol {w} _ {t}) + \frac {L}{2} \| \boldsymbol {w} _ {t + 1} - \boldsymbol {w} _ {t} \| ^ {2} = - \eta \boldsymbol {g} _ {t} ^ {\top} \left(\sum_ {i} \frac {1}{B} \tilde {\boldsymbol {g}} _ {t, i}\right) + \frac {L \eta^ {2}}{2} \left\| \sum_ {i} \frac {1}{B} \tilde {\boldsymbol {g}} _ {t, i} \right\| ^ {2}
$$
The expected improvement at one iteration is
$$
\begin{array}{l} \mathbb {E} \left(\mathcal {L} _ {t + 1} - \mathcal {L} _ {t} \mid \boldsymbol {w} _ {t}\right) \leq - \eta \boldsymbol {g} _ {t} ^ {\top} \mathbb {E} \tilde {\boldsymbol {g}} _ {t, i} + \frac {L \eta^ {2}}{2} \mathbb {E} \| \sum_ {i} \frac {1}{B} \tilde {\boldsymbol {g}} _ {t, i} \| ^ {2} \tag {D.1} \\ \leq - \eta \| \boldsymbol {g} _ {t} \| ^ {2} + \frac {L \eta^ {2}}{2} \left(\| \boldsymbol {g} _ {t} \| ^ {2} + \frac {\xi^ {2}}{B}\right) \\ \end{array}
$$
Now we extend the expectation over randomness in the trajectory, and perform a telescoping sum over the iterations
$$
\mathcal {L} _ {0} - \mathcal {L} _ {*} \geq \mathcal {L} _ {0} - \mathbb {E} \mathcal {L} _ {T} = \sum_ {t} \mathbb {E} \left(\mathcal {L} _ {t} - \mathcal {L} _ {t + 1}\right) \geq \left(\eta - \frac {L \eta^ {2}}{2}\right) \mathbb {E} \left(\sum_ {t} \| \boldsymbol {g} _ {t} \| ^ {2}\right) - \frac {T L \eta^ {2} \xi^ {2}}{2 B}
$$
Notice that we do not need the symmetry assumption in Assumption 5.3 in the non-DP SGD analysis.
We apply the same learning rate as in [5],  $\eta = \frac{1}{L\sqrt{T}}$
$$
2 \left(\mathcal {L} _ {0} - \mathcal {L} _ {*}\right) \geq \left(\frac {2}{L \sqrt {T}} - \frac {1}{L T}\right) \mathbb {E} \left(\sum_ {t} \| \boldsymbol {g} _ {t} \| ^ {2}\right) - \frac {T \xi^ {2}}{B L T} \geq \frac {\sqrt {T}}{L} \mathbb {E} \left(\frac {1}{T} \sum_ {t} \| \boldsymbol {g} _ {t} \| ^ {2}\right) - \frac {\xi^ {2}}{B L T}
$$
$$
\min  _ {t} \mathbb {E} \left(\| \boldsymbol {g} _ {t} \| ^ {2}\right) \leq \mathbb {E} \left(\frac {1}{T} \sum_ {t} \| \boldsymbol {g} _ {t} \| ^ {2}\right) \leq \frac {1}{\sqrt {T}} \left[ 2 \left(\mathcal {L} _ {0} - \mathcal {L} _ {*}\right) L + \frac {\xi^ {2}}{B} \right]
$$
Using the Jensen's inequality, we can have
$$
\min _ {t} \mathbb {E} \left(\| \boldsymbol {g} _ {t} \|\right) \leq \frac {1}{T ^ {1 / 4}} \sqrt {2 (\mathcal {L} _ {0} - \mathcal {L} _ {*}) L + \frac {\xi^ {2}}{B}}
$$
E Auxiliary proofs
E.1 Proof of Theorem 7
Proof. We first show  $\frac{df(c, S; \Gamma)}{dS} < 0$  for all  $0 < c < 1, \Gamma > 0$  and  $S > 0$ , as visualized in the left plot of Figure 10. We can explicitly write down the derivative, by WolframAlpha
$$
\frac {d f (c , S ; \Gamma)}{d S} = \frac {- (A \Gamma^ {2} + B \Gamma + C)}{\sqrt {S ^ {2} - 2 c S + 1} \sqrt {S ^ {2} + 2 c S + 1} (\Gamma + \sqrt {S ^ {2} - 2 c S + 1}) ^ {2} (\Gamma + \sqrt {S ^ {2} + 2 c S + 1}) ^ {2}} \tag {E.1}
$$
$$
A (c, S) = \sqrt {S ^ {2} + 2 c S + 1} \left(3 c ^ {2} S - 2 c (S ^ {2} + 1) + S\right) + \sqrt {S ^ {2} - 2 c S + 1} \left(3 c ^ {2} S + 2 c (S ^ {2} + 1) + S\right)
$$
$$
B (c, S) = 4 S \left[ \left(S ^ {2} + 1\right) \left(1 - c ^ {2}\right) + c ^ {2} \sqrt {S ^ {2} + 2 c S + 1} \sqrt {S ^ {2} - 2 c S + 1} \right]
$$
$$
C (c, S) = (1 - c ^ {2}) S \left[ (S ^ {2} - 2 c S + 1) ^ {3 / 2} + (S ^ {2} + 2 c S + 1) ^ {3 / 2} \right]
$$
It is obvious that, since  $c < 1$ ,
$$
S ^ {2} \pm 2 c S + 1 > S ^ {2} \pm 2 c S + c ^ {2} = (S \pm c) ^ {2} \geq 0. \tag {E.2}
$$
From (E.2), the denominator in (E.1) is positive and it suffices to show  $A\Gamma^2 + B\Gamma + C > 0$  for all  $0 < c < 1$  and  $S > 0$ , in order to show  $\frac{df}{dS} < 0$ .
Also from (E.2), we can easily see  $B(c, S) > 0$  and  $C(c, S) > 0$ . We will show that  $A(c, S) > 0$  in Lemma E.1, after very heavy algebraic computation.
Now we can claim that  $A\Gamma^2 + B\Gamma + C > 0$  by Fact E.3, and complete the proof of the first statement.
To further see that  $\min_c f(c, S; \Gamma)$  is decreasing in  $S$ , let us denote  $c^*(x; \Gamma) := \arg \min_{c \in [0,1]} f(c, x; \Gamma)$ . Then considering  $S < S'$ , we prove the second statement by observing
$$
\min  _ {c} f (c, S; \Gamma) = f \left(c ^ {*} (S; \Gamma), S; \Gamma\right) > f \left(c ^ {*} (S; \Gamma), S ^ {\prime}; \Gamma\right) \geq \min  _ {c} f (c, S ^ {\prime}; \Gamma).
$$
This statement is also visualized in the right plot of Figure 10.
We next show  $\frac{df(c, S; \Gamma)}{dc} < 0$  for all  $0 < c < 1$ ,  $\Gamma > 0$  and  $S > 1$ . We can explicitly write down the derivative, by WolframAlpha
$$
\frac {d f (c , S ; \Gamma)}{d c} = \frac {- S \left(A ^ {\prime} \Gamma^ {2} + B ^ {\prime} \Gamma + C ^ {\prime}\right)}{\sqrt {S ^ {2} - 2 c S + 1} \sqrt {S ^ {2} + 2 c S + 1} (\Gamma + \sqrt {S ^ {2} - 2 c S + 1}) ^ {2} (\Gamma + \sqrt {S ^ {2} + 2 c S + 1}) ^ {2}} \tag {E.3}
$$
$$
A ^ {\prime} (c, S) = \left[ \left(S ^ {2} + 3 c S + 2\right) \sqrt {S ^ {2} - 2 c S + 1} - \left(S ^ {2} - 3 c S + 2\right) \sqrt {S ^ {2} + 2 c S + 1} \right]
$$
$$
B ^ {\prime} (c, S) = 4 S c \left[ \sqrt {S ^ {2} + 2 c S + 1} \sqrt {S ^ {2} - 2 c S + 1} + \left(S ^ {2} - 1\right) \right]
$$
$$
C ^ {\prime} (c, S) = S \left[ (c + S) (S ^ {2} - 2 c S + 1) ^ {3 / 2} + (c - S) (S ^ {2} + 2 c S + 1) ^ {3 / 2} \right]
$$
Clearly  $B'(c, S) > 0$  and  $C'(c, S) > 0$ , since  $S^2 + 2cS + 1 > S^2 - 2cS + c^2 = (S - c)^2 \geq 0$ . And we will show  $A'(c, S) > 0$  in Lemma E.2, after some algebra.
We again claim that  $A' \Gamma^2 + B' \Gamma + C' > 0$  by Fact E.3, which guarantees that the numerator in (E.3) is negative and that  $\frac{df}{dc} < 0$ . This is visualized in Figure 11.
Figure 10: Visualization of  $f(0.5, S, \Gamma)$  (left) and  $\min_{0 \leq c \leq 1} f(c, S, \Gamma)$  over  $S > 0$ .
Figure 11: Visualization of  $f(c,0.8,\Gamma)$  (left) and  $f(c,2,\Gamma)$  over  $0\leq c\leq 1$ .
E.2 Proof of Lemma E.1
Lemma E.1. For all  $0 < c < 1$  and  $S > 0$ ,
$$
A := \sqrt {S ^ {2} + 2 c S + 1} \left(3 c ^ {2} S - 2 c (S ^ {2} + 1) + S\right) + \sqrt {S ^ {2} - 2 c S + 1} \left(3 c ^ {2} S + 2 c (S ^ {2} + 1) + S\right) > 0.
$$
Proof. We prove by contradiction. Suppose
$$
\sqrt {S ^ {2} + 2 c S + 1} \left(3 c ^ {2} S - 2 c (S ^ {2} + 1) + S\right) + \sqrt {S ^ {2} - 2 c S + 1} \left(3 c ^ {2} S + 2 c (S ^ {2} + 1) + S\right) <   0.
$$
$$
0 <   \sqrt {S ^ {2} - 2 c S + 1} \left(3 c ^ {2} S + 2 c (S ^ {2} + 1) + S\right) <   - \sqrt {S ^ {2} + 2 c S + 1} \left(3 c ^ {2} S - 2 c (S ^ {2} + 1) + S\right).
$$
where the first inequality comes from  $S^2 - 2cS + 1 > S^2 - 2cS + c^2 = (S - c)^2 \geq 0$ .
Squaring everything gives
$$
\left(S ^ {2} - 2 c S + 1\right) \left(3 c ^ {2} S + 2 c \left(S ^ {2} + 1\right) + S\right) ^ {2} <   \left(S ^ {2} + 2 c S + 1\right) \left(3 c ^ {2} S - 2 c \left(S ^ {2} + 1\right) + S\right) ^ {2}.
$$
Taking the difference gives
$$
4 c S \left(2 + 3 S ^ {2} - 9 c ^ {4} S ^ {2} + 2 S ^ {4} + 2 c ^ {2} \left(1 - S ^ {2} + S ^ {4}\right)\right) <   0
$$
Given that  $c > 0, S > 0$ , we have
$$
2 + 3 S ^ {2} - 9 c ^ {4} S ^ {2} + 2 S ^ {4} + 2 c ^ {2} \left(1 - S ^ {2} + S ^ {4}\right) <   0
$$
Denoting  $X \coloneqq S^2$  and viewing the above as a quadratic polynomial of  $X$ , we have
$$
\underbrace {(2 c ^ {2} + 2) X ^ {2} + (3 - 2 c ^ {2} - 9 c ^ {4}) X + (2 c ^ {2} + 2)} _ {\text {①}} <   0
$$
Using the closed-form minimizer of quadratic polynomial ①, after some heavy algebra, one can check the minimum of ① is
$$
\frac {(1 + 3 c ^ {2}) ^ {2} (1 - c ^ {2}) (7 + 9 c ^ {2})}{8 (1 + c ^ {2})}
$$
which is clearly positive. Contradiction!
E.3 Proof of Lemma E.2
Lemma E.2. For all  $0 < c < 1$  and  $S > 1$ ,
$$
(S ^ {2} + 3 c S + 2) \sqrt {S ^ {2} - 2 c S + 1} - (S ^ {2} - 3 c S + 2) \sqrt {S ^ {2} + 2 c S + 1} > 0.
$$
Proof. Notice that  $(S^2 + 3cS + 2) > S^2 + 2 > 0$  and  $\sqrt{S^2 \pm 2cS + 1} > 0$ . Therefore if  $S^2 - 3cS + 2 \leq 0$ , we are done.
Otherwise, we prove by contradiction and suppose
$$
0 <   (S ^ {2} + 3 c S + 2) \sqrt {S ^ {2} - 2 c S + 1} <   (S ^ {2} - 3 c S + 2) \sqrt {S ^ {2} + 2 c S + 1}.
$$
under the condition that  $S^2 - 3cS + 2 > 0$ .
$$
(S ^ {2} + 3 c S + 2) ^ {2} (S ^ {2} - 2 c S + 1) <   (S ^ {2} - 3 c S + 2) ^ {2} (S ^ {2} + 2 c S + 1).
$$
$$
c S \left(8 + 2 0 S ^ {2} - 3 6 c ^ {2} S ^ {2} + 8 S ^ {4}\right) <   0
$$
$$
2 + 5 S ^ {2} - 9 c ^ {2} S ^ {2} + 2 S ^ {4} <   0
$$
Denoting  $X \coloneqq S^2$  and viewing the above as a quadratic polynomial of  $X$ , we have, for  $X > 1$
$$
\underbrace {2 X ^ {2} + (5 - 9 c ^ {2}) X + 2} _ {\text {②}} <   0
$$
The closed-form minimizer of quadratic polynomial ② is  $\frac{(9c^2 - 5)}{4}$ . Given that  $0 < c < 1$ , we must have  $-\frac{5}{4} < \frac{9c^2 - 5}{4} < 1$ . Hence the minimizer is not within the feasible domain  $(1, \infty)$  of  $X$ . Thus the minimum of ② is achieved with  $X = 1$  at  $9(1 - c^2)$ . This is positive. Contradiction!
E.4 Proof of Fact E.3
Fact E.3. For a quadratic polynomial  $Ax^2 + Bx + C$  with  $A, B, C > 0$ , the minimum value on the domain  $x \geq 0$  is  $C$ , at  $x = 0$ . Therefore  $Ax^2 + Bx + C > 0$ .
Proof. Since  $A > 0$ , the quadratic polynomial is convex and increasing on the domain  $x > -\frac{B}{2A}$ . Since  $B > 0$  as well, we know  $-\frac{B}{2A} < 0$  and hence the quadratic polynomial is strictly increasing on  $x > 0$ . Therefore the minimum value is achieved when  $x = 0$ , and we obtain  $Ax^2 + Bx + C \geq C > 0$  for all  $x \geq 0$ .
E.5 Assumption of symmetric gradient noise
We show that Assumption 5.3 is actually relaxed from and less strict than the assumptions used in the non-DP literature. In words, Assumption 5.3 allows our DP convergence to be comparable to the standard convergence (as in Theorem 9), because our assumption does not enforce extra constraint.
In standard non-DP analysis [48, 66, 12, 73], the mini-batch gradient is assumed to be an unbiased estimate of the oracle gradient  $\pmb{g}_t = \frac{\partial\mathcal{L}}{\partial\pmb{w}}$ :
$$
\frac {1}{B} \sum_ {i = 1} ^ {B} \tilde {\boldsymbol {g}} _ {t, i} \sim \boldsymbol {g} _ {t} + \boldsymbol {\xi} (\boldsymbol {w})
$$
and  $\xi$  is the random gradient noise with  $\xi \sim N(0, \Sigma(w) / B)$ . Since this assumption holds for any batch size  $B$ , we can set  $B = 1$  to recover the per-sample gradient noise:  $\xi = \tilde{g}_{t,i} - g_t$  is i.i.d. and symmetric because a zero-mean Gaussian is symmetric.
In fact, we can further relax our Assumption 5.3: besides assuming the central symmetry, the same proof of convergence will follow if we instead assume the mirror symmetry about the hyperplane normal to  $\pmb{g}_t$ , that is  $\{\pmb{v}:\pmb{g}_t^\top \pmb{v} = 0\}$ .
F Examples of lazy regions
F.1 Balanced binary classification
We describe the data generation in Section 3.3. The label is uniformly  $\pm 1$ , that is  $\mathbb{P}(y_i = +1) = \mathbb{P}(y_i = -1) = 0.5$ . We have 10000 positive and negative samples  $x_i \sim \mathcal{N}(y_i, 1)$ . We consider a logistic regression model  $\mathbb{P}(Y = y|x) = \mathbb{I}(y = 1) \cdot \text{Sigmoid}(x + \theta) + \mathbb{I}(y = -1) \cdot (1 - \text{Sigmoid}(x + \theta)) = \frac{1}{1 + e^{-y(\theta + x)}}$ , where  $\theta \in \mathbb{R}$  is the intercept. The gradient with respect to this only trainable parameter is  $\frac{\partial \mathcal{L}_i}{\partial \theta} = -y\left(1 - \frac{1}{1 + e^{-y(\theta + x)}}\right)$ . We set the clipping threshold  $R = 0.01$  and the stability constant  $\gamma = 0.01$ .
F.2 Mean estimation on Gaussian mixture data
We also observe the lazy region issue in the mean estimation problem  $\min_{\theta} \frac{1}{2} \| \theta - x_i \|^2$ . Here  $\mathbb{P}(x_i \sim \mathcal{N}(4, 1)) = \mathbb{P}(x_i \sim \mathcal{N}(4, 1)) = 0.5$ . We have 10000 samples from each Gaussian distribution. The regular minimum is clearly  $\sum_{i} x_i \to 0$ , where the regular gradient and AUTO-S clipped gradient vanish. Yet both AUTO-V and Abadi's clipping lose motivation to update the mean estimator on the interval  $(-1, 1)$ . We set the clipping threshold  $R = 0.01$  and the stability constant  $\gamma = 0.1$ .
Figure 12: Scalar gradient  $\frac{\partial\mathcal{L}}{\partial\theta}$  at each  $\theta$
G Experiments settings
G.1 Image classification settings
We give the experiments settings for computer vision tasks in Table 1.
- MNIST: We use the network architecture from [56, 68, 64], with 40 epochs, 512 batch size, 0.5 learning rate (or 0.005 non-DP learning rate), 0.1 clipping threshold, DP-SGD with 0.9 momentum, and without pretraining. This setting is the same as [68].
- FashionMNIST: We use the same network architecture as MNIST, with 40 epochs, 2048 batch size, 4 learning rate (or 0.04 non-DP learning rate), DP-SGD with 0.9 momentum, and without pretraining. This setting is the same as [68].
- CIFAR10 pretrained: We use the SimCLR model from [13]<sup>7</sup>, with 50 epochs, 1024 batch size, 4 learning rate (or 0.04 non-DP learning rate), 0.1 clipping threshold, and DP-SGD with 0.9 momentum. The SimCLR model is pretrained on unlabeled ImageNet dataset. After pretraining, we obtain a feature of dimension 4096 on which a linear classifier is trained privately. This setting is the same as [68].
- ImageNette: We use the ResNet9 (2.5 million parameters) with Mish activation function [52]. We set 50 epochs, 1000 batch size, 0.0005 learning rate (or 0.000005 non-DP learning rate), 1.5 clipping threshold, and use DP-NAdam, without pretraining. This setting is the same as [36] except we did not apply the learning rate decaying scheduler.
- CelebA (Smiling and Male and Multi-label) We use the same ResNet9 as above, with 10 epochs, 500 batch size, 0.001 DP learning rate (or 0.00001 non-DP learning rate), 0.1 clipping threshold, and use DP-Adam, without pretraining. We use the labels 'Smiling' and 'Male' for two binary classification tasks, with cross-entropy loss. For the multi-label task uses a scalar loss by summing up the 40 binary cross-entropy losses from each label.
We refer the code for MNIST, FashionMNIST, CIFAR10, CIFAR10 pretrained to https://github.com/ftramer/Handcrafted-DP by [68]. ResNet9 can be found in https://github.com/cbenitez81/Resnet9.
See implementation in https://github.com/google-research/simblr.
Throughout all experiments, we do not apply tricks such as random data augmentation (single or multiple times [17]), weight standardization [61], or parameter averaging [60].
G.2 Sentence classification settings
We experiment on five datasets in Table 2 and Table 3.
- MNLI(m) MNLI-matched, the matched validation and test splits from Multi-Genre Natural Language Inference Corpus.
- MNLI(mm) MNLI-mismatched, the matched validation and test splits from Multi-Genre Natural Language Inference Corpus.
- QQP The Quora Question Pairs2 dataset.
- QNLI The Stanford Question Answering dataset.
- SST2 The Stanford Sentiment Treebank dataset.
The datasets are processed and loaded from Huggingface [39], as described in https://huggingface.co/datasets/glue. We follow the same setup as [78] and [41]. We refer the interested readers to Appendix G,H,I,K,N of [41] for more details.
We emphasize that our automatic clipping uses exactly the same hyperparameters as the Abadi's clipping in [41], which is released in their Private-Transformers library<sup>8</sup>.