SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} V (\pmb {x}) = - \frac {1}{8} \eta I \| \nabla f (\pmb {x}) \| ^ {2} + \left(2 \Gamma_ {1} \eta^ {2} I ^ {2} \left(2 \sigma + \frac {\gamma}{\eta}\right) + \frac {2 1 6 B L _ {1} \sigma^ {2} \eta^ {2} I \log \frac {1}{\delta}}{S}\right) \| \nabla f (\pmb {x}) \| + \\ 9 6 \Gamma_ {1} ^ {3} \eta^ {4} I ^ {4} \left(2 \sigma + \frac {\gamma}{\eta}\right) ^ {2} + \frac {2 1 6 A L _ {0} \sigma^ {2} \eta^ {2} I \log \frac {1}{\delta}}{S}, \\ \end{array}
$$
Lemma 12 gives us
$$
f \left(\bar {\boldsymbol {x}} _ {R}\right) - f \left(\bar {\boldsymbol {x}} _ {0}\right) \leq \sum_ {r = 0} ^ {R - 1} \left[ \mathbb {1} _ {\bar {\mathcal {A}} _ {r}} U \left(\bar {\boldsymbol {x}} _ {r}\right) + \mathbb {1} _ {\mathcal {A} _ {r}} V \left(\bar {\boldsymbol {x}} _ {r}\right) \right] + 1 2 \eta \sigma^ {2} \log \frac {1}{\delta} + \Gamma_ {2} \eta I \log \frac {1}{\delta}. \tag {40}
$$
We will bound each of  $\mathbb{1}_{\bar{\mathcal{A}}_r}U(\bar{\boldsymbol{x}}_r)$  and  $\mathbb{1}_{\mathcal{A}_r}V(\bar{\boldsymbol{x}}_r)$  under our choices of  $\gamma$  and  $\eta$ .
To bound  $U(\bar{\mathbf{x}}_r)$ , notice
$$
\begin{array}{l} - \frac {3}{4} + 2 B L _ {1} \rho \gamma I + \frac {1}{2} B L _ {1} \gamma I \leq - \frac {3}{4} + \frac {5}{2} B L _ {1} \rho \gamma I \\ \leq - \frac {3}{4} + \frac {5}{2} \eta I B L _ {1} \rho \frac {\gamma}{\eta} \\ = - \frac {3}{4} + \frac {5}{2} \eta I B L _ {1} \rho \left(7 2 \sigma + \frac {A L _ {0}}{B L _ {1} \rho}\right) \\ = - \frac {3}{4} + \frac {5}{2} \eta I (7 2 B L _ {1} \rho \sigma + A L _ {0}) \\ = - \frac {3}{4} + 4 5 \Gamma_ {1} \eta I \\ \leq - \frac {3}{4} + \frac {1}{2} \\ \leq - \frac {1}{4}, \\ \end{array}
$$
$$
\begin{array}{l} - \frac {1}{8} \frac {\gamma}{\eta} + 6 \sigma + 2 A L _ {0} \gamma I + \frac {5}{2} B L _ {1} \kappa \gamma I = - \frac {1}{8} \frac {\gamma}{\eta} + 6 \sigma + \eta I \frac {\gamma}{\eta} \left(2 A L _ {0} + \frac {5}{2} B L _ {1} \kappa\right) \\ \leq - \frac {1}{8} \frac {\gamma}{\eta} + 6 \sigma + 2 \Gamma_ {1} \eta I \frac {\gamma}{\eta} \\ \leq \left(- \frac {1}{8} + \frac {1}{3 0}\right) \frac {\gamma}{\eta} + 6 \sigma \\ \leq - \frac {1}{1 2} \left(7 2 \sigma + \frac {A L _ {0}}{B L _ {1} \rho}\right) + 6 \sigma \\ \leq 0. \\ \end{array}
$$
$$
U (\bar {\boldsymbol {x}} _ {r}) = - \frac {1}{4} \gamma I \| \nabla f (\bar {\boldsymbol {x}} _ {r}) \| \leq - \frac {1}{4} \epsilon \eta I \| \nabla f (\bar {\boldsymbol {x}} _ {r}) \|,
$$
where the last inequality holds due to the fact  $\epsilon \leq \frac{AL_0}{16BL_1\rho} \leq 72\sigma + \frac{AL_0}{BL_1\rho} = \frac{\gamma}{\eta}$ . To bound  $V(\bar{\mathbf{x}}_r)$ , notice
$$
2 \Gamma_ {1} \eta^ {2} I ^ {2} \left(2 \sigma + \frac {\gamma}{\eta}\right) \leq \frac {1}{1 6} \epsilon \eta I,
$$
since  $\eta \leq \frac{\epsilon}{32\Gamma_1I\left(74\sigma + \frac{AL_0}{BL_1\rho}\right)} = \frac{\epsilon}{32\Gamma_1I\left(2\sigma + \frac{\gamma}{\eta}\right)}$ , and
$$
\begin{array}{l} \frac {2 1 6 B L _ {1} \sigma^ {2} \eta^ {2} I \log \frac {1}{\delta}}{S} = \frac {B L _ {1}}{A L _ {0}} \eta I \frac {2 1 6 A L _ {0} \sigma^ {2} \eta \log \frac {1}{\delta}}{S} \\ \leq \frac {B L _ {1}}{A L _ {0}} \epsilon^ {2} \eta I \\ \leq \frac {1}{1 6} \epsilon \eta I, \\ \end{array}
$$
where the last line uses  $\epsilon \leq \frac{AL_0}{16BL_1\rho} \leq \frac{AL_0}{16BL_1}$ . Lastly,
$$
\begin{array}{l} 9 6 \Gamma_ {1} ^ {3} \eta^ {4} I ^ {4} \left(2 \sigma + \frac {\gamma}{\eta}\right) ^ {2} + \frac {2 1 6 A L _ {0} \sigma^ {2} \eta^ {2} I \log \frac {1}{\delta}}{S} \\ \leq \eta I \left(9 6 \Gamma_ {1} ^ {3} \eta^ {3} I ^ {3} \left(2 \sigma + \frac {\gamma}{\eta}\right) ^ {2} + \frac {2 1 6 A L _ {0} \sigma^ {2} \eta \log \frac {1}{\delta}}{S}\right) \\ \leq \eta I \left(\frac {8}{5} \epsilon^ {2} + \epsilon^ {2}\right) \\ \leq 3 \epsilon^ {2} \eta I. \\ \end{array}
$$
$$
\begin{array}{l} V \left(\bar {\boldsymbol {x}} _ {r}\right) = - \frac {1}{8} \eta I \| \nabla f \left(\bar {\boldsymbol {x}} _ {r}\right) \| ^ {2} + \frac {1}{8} \epsilon \eta I \| \nabla f \left(\bar {\boldsymbol {x}} _ {r}\right) \| + 3 \epsilon^ {2} \eta I \\ \leq - \frac {1}{8} \epsilon \eta I \| \nabla f (\bar {\boldsymbol {x}} _ {r}) \| + 4 \epsilon^ {2} \eta I, \\ \end{array}
$$
where the last line came from the inequality  $x^{2} \geq 2ax + a^{2}$  with  $x = \| \nabla f(\bar{\mathbf{x}}_r)\|$  and  $a = \epsilon$ . Combining the bounds of  $U(\bar{\mathbf{x}}_r)$  and  $V(\bar{\mathbf{x}}_r)$ ,
$$
\max  \left\{U (\bar {\boldsymbol {x}} _ {r}), V (\bar {\boldsymbol {x}} _ {r}) \right\} \leq - \frac {1}{8} \epsilon \eta I \| \nabla f (\bar {\boldsymbol {x}} _ {r}) \| + 4 \epsilon^ {2} \eta I.
$$
Plugging this into Equation 40 yields
$$
f (\bar {\boldsymbol {x}} _ {R}) - f (\bar {\boldsymbol {x}} _ {0}) \leq - \frac {1}{8} \epsilon \eta I \sum_ {r = 0} ^ {R - 1} \| \nabla f (\bar {\boldsymbol {x}} _ {r}) \| + 4 \epsilon^ {2} \eta R I + 1 2 \eta \sigma^ {2} \log \frac {1}{\delta} + \Gamma_ {2} \eta I \log \frac {1}{\delta},
$$
and by rearranging we have
$$
\begin{array}{l} \frac {1}{R} \sum_ {r = 0} ^ {R - 1} \| \nabla f (\bar {\boldsymbol {x}} _ {r}) \| \leq \frac {8 (f (\bar {\boldsymbol {x}} _ {0}) - f (\bar {\boldsymbol {x}} _ {R}))}{\epsilon \eta R I} + 3 2 \epsilon + \frac {9 6 \sigma^ {2} \log \frac {1}{\delta}}{\epsilon R I} + \frac {8 \Gamma_ {2} \log \frac {1}{\delta}}{\epsilon R} \\ \leq \frac {8 \Delta}{\epsilon \eta R I} + 3 2 \epsilon + \frac {9 6 \sigma^ {2} \log \frac {1}{\delta}}{\epsilon R I} + \frac {8 \Gamma_ {2} \log \frac {1}{\delta}}{\epsilon R} \\ \leq 3 3 \epsilon + \frac {9 6 \sigma^ {2} \log \frac {1}{\delta}}{\epsilon R I} + \frac {8 \Gamma_ {2} \log \frac {1}{\delta}}{\epsilon R}, \tag {41} \\ \end{array}
$$
where the last line comes from  $R \geq \frac{8\Delta}{\epsilon^2\eta I}$ . Finally,  $\eta \leq \frac{\Delta}{\log\frac{1}{\delta}}\min \left\{\frac{1}{12\sigma^2},\frac{1}{\Gamma_2I}\right\}$  implies
$$
R \geq \frac {8 \Delta}{\epsilon^ {2} \eta I} \geq \frac {9 6 \sigma^ {2} \log \frac {1}{\delta}}{\epsilon^ {2} I},
$$
$$
R \geq \frac {8 \Delta}{\epsilon^ {2} \eta I} \geq \frac {8 \Gamma_ {2} \log \frac {1}{\delta}}{\epsilon^ {2}},
$$
$$
\frac {9 6 \sigma^ {2} \log \frac {1}{\delta}}{\epsilon R I} + \frac {8 \Gamma_ {2} \log \frac {1}{\delta}}{\epsilon R} \leq 2 \epsilon .
$$
Plugging into Equation 41 gives the result.
A.6 Proof of Corollary 1
Corollary 1 restated. If, under the setting of Theorem 1, we additionally choose
$$
\eta = \min \left\{\frac {1}{9 0 (K + 1) \Gamma_ {1} I}, \frac {\epsilon}{3 2 \Gamma_ {1} I \left(7 4 \sigma + \frac {A L _ {0}}{B L _ {1} \rho}\right)}, \frac {S \epsilon^ {2}}{2 1 6 A L _ {0} \sigma^ {2} \log \frac {1}{\delta}}, \frac {\Delta}{\log \frac {1}{\delta}} \min \left\{\frac {1}{1 2 \sigma^ {2}}, \frac {1}{\Gamma_ {2} I} \right\} \right\},
$$
$$
I \leq \frac {2 7 A L _ {0} \sigma^ {2} \log \frac {1}{\delta}}{4 \epsilon \Gamma_ {1} S \left(7 4 \sigma + \frac {A L _ {0}}{B L _ {1} \rho}\right)},
$$
$$
\epsilon \leq \min  \left\{\frac {1 6 \left(7 4 \sigma + \frac {A L _ {0}}{B L _ {1} \rho}\right)}{4 5 (K + 1)}, \sqrt {\frac {1 8 \Delta A L _ {0}}{S}}, \frac {3 2 \Delta \Gamma_ {1} \left(7 4 \sigma + \frac {A L _ {0}}{B L _ {1} \rho}\right)}{\Gamma_ {2} \log \frac {1}{\delta}} \right\},
$$
then with probability  $1 - 14\delta$ , Algorithm 1 will reach an  $\epsilon$ -stationary point with iteration complexity
$$
R I = O \left(\frac {\Delta L _ {0} \sigma^ {2} \log \frac {1}{\delta}}{S \epsilon^ {4}}\right).
$$
Proof. From the condition on  $\epsilon$ ,
$$
\epsilon \leq \frac {1 6 \left(7 4 \sigma + \frac {A L _ {0}}{B L _ {1} \rho}\right)}{4 5 (K + 1)}
$$
$$
\frac {\epsilon}{3 2 \Gamma_ {1} I \left(7 4 \sigma + \frac {A L _ {0}}{B L _ {1} \rho}\right)} \leq \frac {1}{9 0 (K + 1) \Gamma_ {1} I}.
$$
From the condition on  $I$
$$
\frac {\epsilon}{3 2 \Gamma_ {1} I \left(7 4 \sigma + \frac {A L _ {0}}{B L _ {1} \rho}\right)} \geq \frac {\epsilon}{3 2 \Gamma_ {1} \left(7 4 \sigma + \frac {A L _ {0}}{B L _ {1} \rho}\right)} \frac {4 \epsilon \Gamma_ {1} S \left(7 4 \sigma + \frac {A L _ {0}}{B L _ {1} \rho}\right)}{2 7 A L _ {0} \sigma^ {2} \log \frac {1}{\delta}} = \frac {S \epsilon^ {2}}{2 1 6 A L _ {0} \sigma^ {2} \log \frac {1}{\delta}}
$$
$$
\frac {S \epsilon^ {2}}{2 1 6 A L _ {0} \sigma^ {2} \log \frac {1}{\delta}} \leq \frac {S}{2 1 6 A L _ {0} \sigma^ {2} \log \frac {1}{\delta}} \frac {1 8 \Delta A L _ {0}}{S} = \frac {\Delta}{1 2 \sigma^ {2} \log \frac {1}{\delta}},
$$
$$
\begin{array}{l} \frac {S \epsilon^ {2}}{2 1 6 A L _ {0} \sigma^ {2} \log \frac {1}{\delta}} \leq \frac {S}{2 1 6 A L _ {0} \sigma^ {2} \log \frac {1}{\delta}} \frac {3 2 \Delta \Gamma_ {1} \left(7 4 \sigma + \frac {A L _ {0}}{B L _ {1} \rho}\right)}{\Gamma_ {2} \log \frac {1}{\delta}} \frac {2 7 A L _ {0} \sigma^ {2} \log \frac {1}{\delta}}{4 \Gamma_ {1} I S \left(7 4 \sigma + \frac {A L _ {0}}{B L _ {1} \rho}\right)} \\ \leq \frac {\Delta}{\Gamma_ {2} I \log \frac {1}{\delta}}. \\ \end{array}
$$
Therefore  $\eta = \frac{S\epsilon^2}{216AL_0\sigma^2\log\frac{1}{\delta}}$ . Choosing  $R = \frac{8\Delta}{\epsilon^2\eta I}$ ,
$$
R I = \frac {8 \Delta}{\epsilon^ {2} \eta} = \frac {1 2 9 6 A \Delta L _ {0} \sigma^ {2} \log \frac {1}{\delta}}{S \epsilon^ {4}} = O \left(\frac {\Delta L _ {0} \sigma^ {2} \log \frac {1}{\delta}}{S \epsilon^ {4}}\right).
$$
B Deferred Proofs of the Lower Bound
Algorithm 2 Clipped Minibatch SGD
1: Initialize  $x_0$
2: for  $r = 0,1,\ldots ,R - 1$  do
3: Sample  $S_r \subset [N]$  uniformly at random such that  $|\mathcal{S}_r| = S$
4:  $\pmb{g}_r = \frac{1}{SI}\sum_{i\in S_r}\sum_{k = 0}^{I - 1}\nabla F_i(\pmb{x}_r,\xi_{r,k}^i)$
5: Update  $\pmb{x}_{r+1} \gets \pmb{x}_r - \min \left(\eta, \frac{\gamma}{\|\pmb{g}_r\|}\right) \pmb{g}_r$
6: end for
Our proof of Theorem 2 analyzes three separate functions satisfying Assumption 1, and concludes that, in order for clipped minibatch SGD to avoid divergence on the first two functions, one must choose parameters  $\gamma, \eta$  such that convergence on the third function is slow. These three functions are separately analyzed in Lemmas 13, 15, and 16. Each lemma assumes a particular value of the initial point  $x_0$ : this assumption can be made without loss of generality, since for the initialization  $x_0$  we may always translate each function to simulate the assumed initialization.
Lemma 13. Let  $\delta \in (0,1)$  and denote  $Q = \left\lfloor \frac{\kappa + (\rho - 1)M}{\kappa + (\rho + 1)M} N\right\rfloor$  Suppose
$$
\rho > \frac {2 + \log (2 - \delta)}{\log (2 - \delta)}, \quad N \geq \frac {(\rho + 1) (1 + \log (2 - \delta))}{(\rho - 1) \log (2 - \delta) - 2}, \quad 1 \leq S \leq \frac {\log (2 - \delta) (Q + 1)}{N - Q + \log (2 - \delta)}, \tag {42}
$$
and  $\frac{\gamma}{\eta} \leq M$ . For any  $0 < \epsilon < M$ , there exists a problem instance  $\{f_i\}_{i=1}^N \in \mathcal{F}(L_0, L_1, M, \kappa, \rho, N)$  such that, with probability at least  $\delta$ , clipped minibatch SGD with parameters  $\gamma, \eta, S$  will generate iterates  $\{\pmb{x}_r\}_{r=0}^R$  with  $\|\nabla f(\pmb{x}_r)\| > \epsilon$  for all  $r$ .
The proof of Lemma 13 will require the following lemma.
Lemma 14 (Lemma 5.3 in [20]). Let  $\{X_{t}\}_{t = 1}^{\infty}$  be a Markov chain over states  $\{i\}_{i = 0}^{\infty}$ , such that  $0$  is an absorbing state, and the transition distribution elsewhere is as follows:
$$
X _ {t + 1} | \{X _ {t} = i \} = \left\{ \begin{array}{l l} i - 1 & w. p. p \\ i + 1 & w. p. 1 - p \end{array} \right.
$$
Define the absorb probabilities  $\alpha_{i}\coloneqq \mathbb{P}(\exists t > 0:X_{t} = 0\mid X_{0} = i)$ , then:
$$
\alpha_ {i} = \left(\frac {p}{1 - p}\right) ^ {i}, \quad \forall i \geq 1
$$
Proof of Lemma 13. Define
$$
\phi = \frac {Q}{N}, b = \frac {1 + \phi}{1 - \phi} M,
$$
$$
v = \frac {3 b}{2 L _ {0}}, \quad g (x) = - \frac {1}{8 v ^ {3}} x ^ {4} + \frac {3}{4 v} x ^ {2}.
$$
$$
\ell_ {1} (x) = \left\{ \begin{array}{l l} M (x + v) - M g (v) & x \leq - v \\ - M g (x) & x \in (- v, v) \\ - M (x - v) - M g (v) & x \geq v \end{array} \right.
$$
$$
\ell_ {2} (x) = \left\{ \begin{array}{l l} - b (x + v) + b g (v) & x \leq - v \\ b g (x) & x \in (- v, v) \\ b (x - v) + b g (v) & x \geq v \end{array} \right.
$$
Consider the problem instance defined by
$$
f _ {i} (x) = \left\{ \begin{array}{l l} \ell_ {1} (x) & 1 \leq i \leq Q \\ \ell_ {2} (x) & Q + 1 \leq i \leq N \end{array} \right.
$$
with global objective
$$
f (x) = \phi \ell_ {1} (x) + (1 - \phi) \ell_ {2} (x) = \left\{ \begin{array}{l l} - M (x + v) + M g (v) & x \leq - v \\ M g (x) & x \in (- v, v) \\ M (x - v) + M g (v) & x \geq v \end{array} \right.
$$
Note that  $b > M$ . For this problem instance, we define the stochastic objective as  $F_{i}(x;\xi) = f_{i}(x)$ , so that the true gradient of each local objective is returned by each gradient query of the algorithm.
By construction, each local objective  $f_{i}$  and the global objective  $f$  is twice continuously differentiable, with gradient bounded by  $M$ . Also, each objective is  $(L_0, L_1)$ -smooth (in fact, they are  $L_0$ -smooth), since for  $|x| \leq v$ ,
$$
\begin{array}{l} | g ^ {\prime \prime} (x) | = \left| - \frac {3}{2 v ^ {3}} x ^ {2} + \frac {3}{2 v} \right| \\ = \frac {3}{2 v} \left| 1 - \frac {x ^ {2}}{v ^ {2}} \right| \\ \leq \frac {3}{2 v} = \frac {L _ {0}}{b}, \\ \end{array}
$$
and for  $|x| \geq v$ ,  $|f_i''(x)| = 0$ . Finally, the collection of objectives satisfies the heterogeneity condition. For  $i \leq Q$
$$
\left| f _ {i} ^ {\prime} (x) \right| = \left| f ^ {\prime} (x) \right| \leq \kappa + \rho \left| f ^ {\prime} (x) \right|,
$$
and for  $i\geq Q + 1$
$$
| f _ {i} ^ {\prime} (x) | = b \frac {| f ^ {\prime} (x) |}{M} \leq \frac {1 + \phi}{1 - \phi} M \frac {| f ^ {\prime} (x) |}{M} \leq (\kappa + \rho M) \frac {| f ^ {\prime} (x) |}{M} \leq \kappa + \rho | f ^ {\prime} (x) |,
$$
where we used
$$
\frac {1 + \phi}{1 - \phi} \leq \frac {1 + \frac {\kappa + (\rho - 1) M}{\kappa + (\rho + 1) M}}{1 - \frac {\kappa + (\rho - 1) M}{\kappa + (\rho + 1) M}} = \frac {2 \kappa + 2 \rho M}{\kappa + (\rho + 1) M} \frac {\kappa + (\rho + 1) M}{2 M} = \frac {\kappa + \rho M}{M},
$$
together with the fact  $\phi \leq \frac{\kappa + (\rho - 1)M}{\kappa + (\rho + 1)M}$ . Therefore  $\{f_i\}_{i = 1}^N\in \mathcal{F}(L_0,L_1,M,\kappa ,\rho ,N)$ .
Now, we analyze the behavior of clipped minibatch SGD on the instance  $\{f_i\}_{i=1}^N$  with initialization  $x_0 = v + 1$ . Let  $r_0$  be the index of the first round (if one exists) in which  $x_r < v$ . For each round  $r$ , define the event  $B_r = \{S_r \subset [Q]\}$ . To compute  $\mathbb{P}(B_r)$ , first notice
$$
\frac {N - Q}{Q - S + 1} \leq \frac {\log (2 - \delta)}{S}.
$$
$$
\mathbb {P} \left(B _ {r}\right) = \prod_ {i = 0} ^ {S - 1} \frac {Q - i}{N - i} \geq \left(\frac {Q - S + 1}{N - S + 1}\right) ^ {S} = \frac {1}{\left(1 + \frac {N - Q}{Q - S + 1}\right) ^ {S}} \geq \frac {1}{\left(1 + \frac {\log (2 - \delta)}{S}\right) ^ {S}} \geq \frac {1}{e ^ {\log (2 - \delta)}} = \frac {1}{2 - \delta},
$$
where we used the inequality  $\left(1 + \frac{a}{x}\right)^x\leq e^a$
Now define the auxiliary sequence  $\{y_r\}_{r=0}^R$  as follows:
$$
y _ {0} = x _ {0}
$$
$$
y _ {r + 1} = \left\{ \begin{array}{l l} y _ {r} + \gamma & \text {i f} B _ {r} \\ y _ {r} - \gamma & \text {o t h e r w i s e} \end{array} \right.
$$
We claim that  $y_{r} \leq x_{r}$  for all  $r \leq r_0$ . For any  $r < r_0$ , if  $B_{r}$  occurs then  $y_{r + 1} - y_{r} = x_{r + 1} - x_{r}$  since in this case  $g_{r} = -M$  and  $x_{r + 1} - x_{r} = \gamma$ , due to the fact that  $\frac{\gamma}{\eta} \leq M$ . If  $B_{r}$  does not occur, then  $x_{r + 1} - x_{r} \geq -\gamma$ , due to the clipping operation, so that  $y_{r + 1} - y_{r} \leq x_{r + 1} - x_{r}$ . Therefore for all  $r \leq r_0$ ,
$$
y _ {r} = y _ {0} + \sum_ {q = 0} ^ {r - 1} y _ {q + 1} - y _ {q} \leq x _ {0} + \sum_ {q = 0} ^ {r - 1} x _ {q + 1} - x _ {q} = x _ {r},
$$
which proves our claim. Denoting  $D = \{|f'(x_r)|\geq M$  for all  $r\geq 1\}$
$$
\begin{array}{l} \mathbb {P} (D) \geq \mathbb {P} \left(x _ {r} \geq - 1 \text {f o r a l l} r \geq 1\right) \\ \geq \mathbb {P} \left(y _ {r} \geq - 1 \text {f o r a l l} r \geq 1\right) \\ \stackrel {(i)} {\geq} 1 - \left(\frac {1 - \mathbb {P} (B _ {r})}{\mathbb {P} (B _ {r})}\right) ^ {\lceil \frac {x _ {0} - v}{\gamma} \rceil} \\ \stackrel {(i i)} {\geq} 1 - \frac {1 - \mathbb {P} (B _ {r})}{\mathbb {P} (B _ {r})} \\ = \frac {2 \mathbb {P} (B _ {r}) - 1}{\mathbb {P} (B _ {r})} \\ \geq \left(\frac {2}{2 - \delta} - 1\right) (2 - \delta) \\ = \delta , \\ \end{array}
$$
where  $(i)$  comes from applying Lemma 14 to the sequence  $y_{r}$  and  $(ii)$  uses  $\lceil \frac{x_0 - v}{\gamma} \rceil \geq 1$  and  $\frac{1 - \mathbb{P}(B_r)}{\mathbb{P}(B_r)} \leq 1 - \delta$ .
The following Lemma uses the same function as in the first half of the proof of Theorem 2 in [8]. However, the sequence of iterates computed by clipped minibatch SGD (our setting) evolves differently than that of GD (their setting).
Lemma 15. Suppose  $\frac{\gamma}{\eta} > M, M > \frac{L_0}{L_1}$ , and  $\eta \geq \frac{2}{L_1M}\left(1 + \log \frac{L_1M}{L_0}\right)$ . For any  $0 < \epsilon < M$ , there exists a problem instance  $\{f_i\}_{i=1}^N \in \mathcal{F}(L_0, L_1, M, \kappa, \rho, N)$  such that clipped minibatch SGD with parameters  $\gamma, \eta$ , and any  $S$  will generate iterates  $\{\pmb{x}_r\}_{r=0}^R$  with  $\|\nabla f(\pmb{x}_r)\| > \epsilon$  for all  $r$ .
Proof. Consider the following function,
$$
f (x) = \left\{ \begin{array}{l l} \frac {L _ {0}}{L _ {1} ^ {2}} \exp \left(- L _ {1} x - 1\right) & x <   - \frac {1}{L _ {1}} \\ \frac {L _ {0}}{2} x ^ {2} + \frac {L _ {0}}{2 L _ {1} ^ {2}} & x \in \left[ - \frac {1}{L _ {1}}, \frac {1}{L _ {1}} \right] \\ \frac {L _ {0}}{L _ {1} ^ {2}} \exp \left(L _ {1} x - 1\right) & x > \frac {1}{L _ {1}} \end{array} \right.
$$
and the problem instance  $f_{i} = f$  for all  $i \in [N]$ , with the initialization  $x_0 = \frac{1}{L_1}\left(1 + \log \frac{ML_1}{L_0}\right)$ . We define the stochastic objective for this problem instance as  $F_{i}(x;\xi) = f_{i}(x)$ , so that the true gradient of each local objective is returned by each gradient query of the algorithm.
Note that  $f$  is bounded from below and  $(L_0, L_1)$ -smooth. Also, since all clients have the same objective and gradients are computed deterministically, this problem instance satisfies Assumption 1. Also, our setting of  $x_0$  is consistent with the definition of  $M$ , since
$$
\begin{array}{l} \left| f ^ {\prime} \left(x _ {0}\right) \right| = \frac {L _ {0}}{L _ {1}} \exp \left(L _ {1} x _ {0} - 1\right) \\ = \frac {L _ {0}}{L _ {1}} \frac {M L _ {1}}{L _ {0}} = M. \\ \end{array}
$$
First, define  $w \coloneqq \frac{1}{L_1} \left( 1 + \log \left( \frac{\gamma}{\eta} \frac{L_1}{L_0} \right) \right)$ . By the condition  $\frac{\gamma}{\eta} > M$ , we know  $w > x_0$ , and by construction  $f'(w) = \frac{\gamma}{\eta}$ . Since  $f'$  is increasing on  $[0, \infty)$ , and decreasing on  $(-\infty, 0]$ , we know  $|f'(w)| \geq \frac{\gamma}{\eta}$  if and only if  $|x| \geq w$ . This means that the clipping operation will be performed when  $|x| \geq w$ , and will not be performed for  $|x| < w$ .
Now we analyze the behavior of clipped minibatch SGD on this problem instance. Suppose that  $|x_0| \leq |x_r| \leq w$  for some  $r \geq 0$ . Then
$$
\eta \geq \frac {2 L _ {1} x _ {0}}{L _ {0} \exp \left(L _ {1} x _ {0} - 1\right)} \geq \frac {2 L _ {1} | x _ {r} |}{L _ {0} \exp \left(L _ {1} | x _ {r} | - 1\right)},
$$
where the first inequality comes from the definition of  $x_0$  and the condition  $\eta \geq \frac{2}{L_1M} \left(1 + \log \frac{L_1M}{L_0}\right)$ , and the second inequality comes from the fact that  $\phi(x) := \frac{2L_1x}{L_0\exp(L_1x - 1)}$  is decreasing on  $\left[\frac{1}{L_1}, \infty\right)$ . Therefore
$$
\eta \left| f ^ {\prime} \left(x _ {r}\right) \right| = \eta \frac {L _ {0}}{L _ {1}} \exp \left(L _ {1} \left| x _ {r} \right| - 1\right) \geq 2 \left| x _ {r} \right|. \tag {43}
$$
Since  $\mathrm{sign}(f'(x_r)) \neq \mathrm{sign}(x_r)$ , this implies  $|x_{r+1}| = |x_r - \eta f'(x_r)| \geq 2|x_r| - |x_r| = |x_r|$ . This shows that the sequence of iterates  $\{x_r\}$  is non-decreasing in absolute value until (if ever) the absolute value exceeds  $w$ . If  $|x_r| \leq w$  for all  $r$ , then we are done, since in this case we have  $|x_r| \geq |x_0|$  and therefore  $|\nabla f(x_r)| \geq |\nabla f(x_0)| = M > \epsilon$  for all  $r$ .
Otherwise, let  $\bar{r}$  be the first index  $r$  for which  $|x_{r}| > w$ . Without loss of generality, assume  $x_{\bar{r}} > 0$  (for  $x_{\bar{r}} < 0$ , the same argument applies with signs reversed). Since  $|x_{\bar{r} -1}|\leq w$
$$
x _ {\bar {r}} = x _ {\bar {r} - 1} - \eta f ^ {\prime} (x _ {\bar {r} - 1}) \leq - w - \eta f ^ {\prime} (- w) = \gamma - w.
$$
$$
x _ {\bar {r} + 1} = x _ {\bar {r}} - \gamma \leq (\gamma - w) - \gamma = - w.
$$
Therefore  $|f'(x_{\bar{r} + 1})| \geq |f'(-w)| = |f'(w)| = \frac{\gamma}{\eta}$ , and so  $x_{\bar{r} + 2} = x_{\bar{r} + 1} + \gamma = x_{\bar{r}}$ . We can then show by induction that  $x_{\bar{r} + 2n} = x_{\bar{r}}$  and  $x_{\bar{r} + 2n + 1} = x_{\bar{r} + 1}$  for all  $n \geq 0$ . Since  $|f'(x_{\bar{r}})|, |f'(x_{\bar{r} + 1})| \geq |f'(w)| > |f'(x_0)| > \epsilon$ , we have  $|f'(x_r)| > \epsilon$  for all  $r$ .
The following lemma is nearly identical to the second half of the proof of Theorem 2 in [8]. We include it here for the sake of completeness.
Lemma 16. Suppose  $\frac{\gamma}{\eta} > M$  and  $\eta < \frac{2}{L_1M}\left(1 + \log \frac{L_1M}{L_0}\right)$ . For any  $0 < \epsilon < M$ , there exists a problem instance  $\{f_i\}_{i=1}^N \in \mathcal{F}(L_0, L_1, M, \kappa, \rho, N)$  such that clipped minibatch SGD with parameters  $\gamma, \eta$ , and any  $S$  requires at least
$$
\frac {L _ {1} M \left(f (x _ {0}) - f ^ {*} - \frac {1 5 \epsilon^ {2}}{1 6 L _ {0}}\right)}{2 \epsilon^ {2} \left(1 + \log \frac {L _ {1} M}{L _ {0}}\right)}
$$
roundsinordertofindan  $\epsilon$  -stationarypoint.
$$
f (x) = \left\{ \begin{array}{l l} - \epsilon x & x <   - \frac {3 \epsilon}{2 L _ {0}} \\ - \frac {L _ {0} ^ {3}}{2 7 \epsilon^ {2}} x ^ {4} + \frac {L _ {0}}{2} x ^ {2} + \frac {9 \epsilon^ {2}}{1 6 L _ {0}} & x \in \left[ - \frac {3 \epsilon}{2 L _ {0}}, \frac {3 \epsilon}{2 L _ {0}} \right] \\ \epsilon x & x > \frac {3 \epsilon}{2 L _ {0}} \end{array} \right.
$$
and the problem instance  $f_{i} = f$  for all  $i \in [N]$ , with the initialization  $x_0 = \frac{3\epsilon}{2L_0} + d$ . We define the stochastic objective for this problem instance as  $F_{i}(x, \xi) = f_{i}(x)$ , so that the true gradient of each local objective is returned by each gradient query of the algorithm.
Note that  $f$  is bounded from below and  $(L_0, L_1)$ -smooth. Also, since all clients have the same objective and gradients are computed deterministically, this problem instance satisfies Assumption 1. Also, our setting of  $x_0$  is consistent with the definition of  $M$ , since  $|f'(x_0)| = \epsilon < M$ .
Now we analyze the behavior of clipped minibatch SGD on this problem instance. Since this problem instance is homogeneous with deterministic gradients, we have  $\| g_r\| \leq \epsilon < M < \frac{\gamma}{\eta}$ , so clipped minibatch SGD will always perform unnormalized updates. Therefore, for  $x_{r}\geq \frac{3\epsilon}{2L_{0}}$
$$
x _ {r + 1} = x _ {r} - \eta g _ {r} = x _ {r} - \eta \epsilon > x _ {r} - \frac {2 \epsilon}{L _ {1} M} \left(1 + \log \frac {L _ {1} M}{L _ {0}}\right).
$$
$$
x _ {r} > x _ {0} - \frac {2 \epsilon r}{L _ {1} M} \left(1 + \log \frac {L _ {1} M}{L _ {0}}\right) = \frac {3 \epsilon}{2 L _ {0}} + d - \frac {2 \epsilon r}{L _ {1} M} \left(1 + \log \frac {L _ {1} M}{L _ {0}}\right)
$$
as long as  $\frac{2\epsilon r}{L_1M}\left(1 + \log \frac{L_1M}{L_0}\right) \leq d$ , or  $r \geq \frac{dL_1M}{2\epsilon\left(1 + \log\frac{L_1M}{L_0}\right)}$ . Notice that
$$
f (x _ {0}) = \frac {3 \epsilon^ {2}}{2 L _ {0}} + d \epsilon ,
$$
$$
d = \frac {1}{\epsilon} \left(f (x _ {0}) - \frac {3 \epsilon^ {2}}{2 L _ {0}}\right) = \frac {1}{\epsilon} \left(f (x _ {0}) - f ^ {*} - \frac {1 5 \epsilon^ {2}}{1 6 L _ {0}}\right).
$$