SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} Y _ {k + 1} - X _ {k + 1} ^ {(n)} = Y _ {k} - X _ {k} ^ {(n)} - \eta \left(\nabla f \left(Y _ {k}, Z _ {k + 1}\right) - \nabla f \left(X _ {k} ^ {(n)}, \hat {Z} _ {k + 1} ^ {(n)}\right)\right) \\ = Y _ {k} - X _ {k} ^ {(n)} - \eta \left(\nabla f \left(Y _ {k}, Z _ {k + 1}\right) - \nabla f \left(X _ {k} ^ {(n)}, Z _ {k + 1}\right) + \nabla f \left(X _ {k} ^ {(n)}, Z _ {k + 1}\right) - \nabla f \left(X _ {k} ^ {(n)}, \hat {Z} _ {k + 1} ^ {(n)}\right)\right) \\ = Y _ {k} - X _ {k} ^ {(n)} - \eta \mathbf {H} _ {k, Z _ {k + 1}} ^ {(n)} \left(Y _ {k} - X _ {k} ^ {(n)}\right) - \eta \left(\nabla f (X _ {k} ^ {(n)}, Z _ {k + 1}) - \nabla f (X _ {k} ^ {(n)}, \hat {Z} _ {k + 1} ^ {(n)})\right) \\ = (\mathrm {I} _ {d} - \eta \mathbf {H} _ {k, Z _ {k + 1}} ^ {(n)}) (Y _ {k} - X _ {k} ^ {(n)}) - \eta \left(\nabla f (X _ {k} ^ {(n)}, Z _ {k + 1}) - \nabla f (X _ {k} ^ {(n)}, \hat {Z} _ {k + 1} ^ {(n)}\right). \\ \end{array}
$$
Now, taking the norm of both sides and using the triangle inequality together with Assumption 1, we can obtain the following:
$$
\begin{array}{l} \| Y _ {k + 1} - X _ {k + 1} ^ {(n)} \| \leq \| I _ {d} - \eta \mathbf {H} _ {k, Z _ {k + 1}} ^ {(n)} \| \| Y _ {k} - X _ {k} ^ {(n)} \| + \eta \| \nabla f (X _ {k} ^ {(n)}, Z _ {k + 1}) - \nabla f (X _ {k} ^ {(n)}, \hat {Z} _ {k + 1} ^ {(n)}) \| \\ \leq \| \mathrm {I} _ {d} - \eta \mathbf {H} _ {k, Z _ {k + 1}} ^ {(n)} \| \| Y _ {k} - X _ {k} ^ {(n)} \| + \eta L (\| X _ {k} ^ {(n)} \| + 1) \| Z _ {k + 1} - \hat {Z} _ {k + 1} ^ {(n)} \|. \\ \end{array}
$$
$$
\begin{array}{l} \| \mathrm {I} _ {d} - \eta \int_ {0} ^ {1} \nabla^ {2} f (y - u (y - x), z) \mathrm {d} u \| = \| \int_ {0} ^ {1} \mathrm {I} _ {d} - \eta \nabla^ {2} f (y - u (y - x), z) \mathrm {d} u \| \\ \leq \int_ {0} ^ {1} \| I _ {d} - \eta \nabla^ {2} f (y - u (y - x), z) \| d u \\ \leq \int_ {0} ^ {1} \sup  _ {x \in \mathbb {R} ^ {d}} \left| \left| I _ {d} - \eta \nabla^ {2} f (x, z) \right| \right| d u \\ = R (z), \tag {43} \\ \end{array}
$$
where again  $R(z) = \sup_{x\in \mathbb{R}^d}\left\| \mathrm{I}_d - \eta \nabla^2 f(x,z)\right\|$ . This allows us to conclude the following:
$$
\left\| Y _ {k + 1} - X _ {k + 1} ^ {(n)} \right\| \leq R \left(Z _ {k + 1}\right) \left\| Y _ {k} - X _ {k} ^ {(n)} \right\| + \eta L \left(\left\| X _ {k} ^ {(n)} \right\| + 1\right) \left\| Z _ {k + 1} - \hat {Z} _ {k + 1} ^ {(n)} \right\|.
$$
Denoting  $B_{k}^{(n)}\coloneqq \| X_{k}^{(n)}\| +1$  , we have:
$$
\begin{array}{l} \left\| Y _ {k + 1} - X _ {k + 1} ^ {(n)} \right\| \leq R \left(Z _ {k + 1}\right) \left\| Y _ {k} - X _ {k} ^ {(n)} \right\| + \eta L B _ {k} ^ {(n)} \left\| Z _ {k + 1} - \hat {Z} _ {k + 1} ^ {(n)} \right\| \\ \leq R \left(Z _ {k + 1}\right) R \left(Z _ {k}\right) \left\| Y _ {k - 1} - X _ {k - 1} ^ {(n)} \right\| + \eta L R \left(Z _ {k + 1}\right) B _ {k - 1} ^ {(n)} \left\| Z _ {k} - \hat {Z} _ {k} ^ {(n)} \right\| + \eta L B _ {k} ^ {(n)} \left\| Z _ {k + 1} - \hat {Z} _ {k + 1} ^ {(n)} \right\| \\ \leq \| Y _ {0} - X _ {0} ^ {(n)} \| \prod_ {i = 1} ^ {k + 1} R (Z _ {i}) + \eta L \sum_ {i = 0} ^ {k} B _ {i} ^ {(n)} \| Z _ {i + 1} - \hat {Z} _ {i + 1} ^ {(n)} \| \prod_ {j = i + 1} ^ {k} R (Z _ {j + 1}), \\ \end{array}
$$
where again, for any sequence  $(a_{i})_{i\geq 0}$ ,  $\prod_{i = j}^{k}a_{i} = 1$ , for  $j > k$ . Furthermore, as  $\beta > 1$  and  $\mathbb{E}[R(Z_1)^\beta] = 1$ , we have:
$$
\delta_ {R} := \mathbb {E} \left[ R \left(Z _ {1}\right) \right] <   1. \tag {44}
$$
Now, we can take the expectation given  $E_{n}$  and use that  $R(Z_{1}),\dots,R(Z_{k + 1})$ , and  $B_0^{(n)},\ldots ,B_k^{(n)}$  are i.i.d. Furthermore, using (43), (44), and proceeding as in the proof of Theorem 4, we can obtain:
$$
\mathbb {E} [ \| Y _ {k + 1} - X _ {k + 1} ^ {(n)} \| ] \leq \eta L (\mathbb {E} [ \| X _ {0} ^ {(n)} \| ] + 1) \mathbb {E} [ \| Z _ {1} - \hat {Z} _ {1} ^ {(n)} \| ] \frac {1}{1 - \delta_ {R}}.
$$
Finally, denoting  $\mathbb{E}[\| X_0^{(n)}\| ] + 1 = c_0$  , we obtain that, with probability larger than  $1 - \epsilon_{n}$  ..
$$
\mathbf {W} _ {1} (\pi , \pi^ {(n)}) \leq c _ {0} L (1 - c _ {\rho} \mathrm {e} ^ {- \rho k}) ^ {- 1} \frac {\eta}{1 - \delta_ {R}} \mathbf {W} _ {1} \left(\mu_ {z}, \mu_ {z} ^ {(n)}\right).
$$
Taking the limit as  $k \to \infty$ , we can conclude the proof of Theorem 6.
C.4 Proof of Theorem 5
Proof. Due to [HM21] we know that there exist  $\alpha, \beta$  such that  $\beta < \alpha$  and:
$$
\lim  _ {t \rightarrow \infty} t ^ {\alpha + \epsilon} \mathbb {P} \left(\| X _ {\infty} \| > t\right) > 0, \quad \text {a n d} \quad \lim  _ {t \rightarrow \infty} t ^ {\beta - \epsilon} \mathbb {P} \left(\| X _ {\infty} \| > t\right) <   + \infty . \tag {45}
$$
Using Lemma 3, we can obtain:
$$
\begin{array}{l} \mathbf {W} _ {1} (\pi , \pi^ {(n)}) \geq \int_ {0} ^ {\infty} | \mathbb {P} (\| X _ {\infty} \| > t) - \mathbb {P} (\| X _ {\infty} ^ {(n)} \| > t) | d t \\ \geq \int_ {t ^ {\prime}} ^ {t ^ {\prime \prime}} | \mathbb {P} (\| X _ {\infty} \| > t) - \mathbb {P} (\| X _ {\infty} ^ {(n)} \| > t) | d t, \\ \end{array}
$$
where we have used that  $\lim_{t\to \infty}\mathbb{P}(\| X_{\infty}\| >t) = 0$  and  $\lim_{t\to \infty}\mathbb{P}(\| X_{\infty}^{(n)}\| >t) = 0$ , in order to select  $t^\prime$  and  $t''$  large enough for the last inequality to hold. Now, using Theorem 2, we can obtain:
$$
\begin{array}{l} \int_ {t ^ {\prime}} ^ {t ^ {\prime \prime}} | \mathbb {P} (\| X _ {\infty} \| > t) - \mathbb {P} (\| X _ {\infty} ^ {(n)} \| > t) | \mathrm {d} t \geq \int_ {t ^ {\prime}} ^ {t ^ {\prime \prime}} \frac {c _ {\alpha}}{t ^ {\alpha + \epsilon}} \mathrm {d} t - \int_ {t ^ {\prime}} ^ {t ^ {\prime \prime}} \mathbb {P} (\| X _ {\infty} ^ {(n)} \| > t) \mathrm {d} t \\ \geq \left(t ^ {\prime \prime} - t ^ {\prime}\right) \frac {c _ {\alpha}}{\left(t ^ {\prime \prime}\right) ^ {\alpha + \epsilon}} - \left(t ^ {\prime \prime} - t ^ {\prime}\right) \mathbb {P} \left(\left\| X _ {\infty} ^ {(n)} \right\| > t ^ {\prime}\right), \tag {46} \\ \end{array}
$$
for some constant  $c_{\alpha}$ . Therefore, by choosing  $t'' = 2t'$ , we can obtain:
$$
\mathbb {P} \left(\| X _ {\infty} ^ {(n)} \| > t ^ {\prime}\right) \geq \frac {c _ {\alpha}}{(2 t ^ {\prime}) ^ {\alpha + \epsilon}} - \frac {1}{t ^ {\prime}} \mathbf {W} _ {1} (\pi , \pi^ {(n)}).
$$
Using similar arguments, we can obtain:
$$
\begin{array}{l} \mathbf {W} _ {1} (\pi , \pi^ {(n)}) \geq \int_ {t ^ {\prime}} ^ {t ^ {\prime \prime}} \mathbb {P} \left(\| X _ {\infty} ^ {(n)} \| > t\right) d t - \int_ {t ^ {\prime}} ^ {t ^ {\prime \prime}} \frac {c _ {\beta}}{t ^ {\beta - \epsilon}} d t \\ \geq (t ^ {\prime \prime} - t ^ {\prime}) \mathbb {P} (\| X _ {\infty} ^ {(n)} \| > t ^ {\prime \prime}) - (t ^ {\prime \prime} - t ^ {\prime}) \frac {c _ {\beta}}{(t ^ {\prime}) ^ {\beta - \epsilon}}. \\ \end{array}
$$
for a constant  $c_{\beta}$ . Choosing  $t'' = 2t'$  and substituting  $\frac{1}{2}\bar{t} = t'$ , we obtain:
$$
\mathbb {P} \left(\| X _ {\infty} ^ {(n)} \| > \bar {t}\right) \leq \frac {2}{\bar {t}} \mathbf {W} _ {1} \left(\pi , \pi^ {(n)}\right) + \frac {2 ^ {\beta - \epsilon} c _ {\beta}}{\bar {t} ^ {\beta - \epsilon}}. \tag {47}
$$
Using Theorem 6 and (46)-(47), we can conclude the proof of Theorem 5.
Figure 8: Estimated tail indices, LeNet, CIFAR-10
Figure 9: Estimated tail indices, AlexNet, MNIST
D Further experimental results
In this section, we present the remaining experimental findings.
D.1 Neural Network Experiments
First, we showcase the remaining plots from Section 4. As outlined in 4.2, we conducted model training for 10,000 iterations, utilizing the cross-entropy loss and employing the MNIST and CIFAR-10 datasets. The learning rates ranged from  $10^{-4}$  to  $10^{-1}$ , while the batch sizes varied between 1 and 10. We employed offline SGD with a subset of the data amounting to  $25\%$ ,  $50\%$ , and  $75\%$ . In Figure 8, we exhibit the estimated tail indices for the LeNet architecture with the CIFAR-10 dataset, while Figure 9 shows the estimated tail indices for the AlexNet architecture implemented with the MNIST dataset.
The inclusion of these plots serves the purpose of completeness, as the overall conclusions remain unchanged; a strong correlation between  $\hat{\alpha}^{(n)}$  and  $\hat{\alpha}$  is exhibited, as well as a notable correlation between  $\hat{\alpha}^{(n)}$ 's and the ratio  $\eta / b$ . These conclusions hold true across all datasets and neural network architectures, providing further substantiation for our theoretical propositions.
D.2 Further tail examination
From [GSZ21], we have that  $\mathbb{P}\left(\| X_{\infty}\| >t\right)\approx t^{-\alpha}$ , so the log-log tail histogram of the stationary distribution should follow a linear line with slope  $-\alpha$  in the tails (for large  $t$ ). We first examine the histograms of the estimated stationary distributions for both the linear regression setting (as in Sec. 4.1) and the NN setting (as in Sec. 4.2), analyzing their behavior as the number of samples  $n$  increases. These are depicted in Figures 10-11. Then, we re-run the experiments and examine the histograms on a log-log scale to see whether the linear slope  $-\alpha$  becomes more apparent. For both cases, as  $n$  increases, we observe that this heavy-tailed phenomenon becomes increasingly apparent, and the tails follow a clearer linear trend. The linear behavior in the tails can be observed in Figures 12-13.
In the entirety of our experiments, encompassing diverse learning rates and layers within both the linear regression and NN settings, a consistent observation emerges: an increase in the number of samples employed in offline SGD leads to a convergence of behavior towards that of online SGD. As previously mentioned in Section 1, while we do not anticipate an exact power-law tailed behavior, it is noteworthy that the tails exhibit progressively more characteristics resembling a power-law
distribution. Specifically, the log-log plots demonstrate a linear trend in the tails. This empirical observation aligns with our theoretical findings, again reinforcing the consistency between the two.
Figure 10: Histograms of the parameter norms for linear regression with Gaussian data
First layer
Third layer
Figure 11: Histograms of the weight norms for FC on MNIST dataset
Figure 12: Log-log histograms of the parameter norms for linear regression with Gaussian data
$\eta = 0.015$
$\eta = 0.03$
$\eta = 0.04$
Figure 13: Log-log histograms of the weight norms for FC on MNIST dataset
$\eta = 0.045$
E Estimator choice justification
Analytical computation of the true tail exponent, even in the linear regression setting, is, to our knowledge, unfortunately not possible. Therefore, an estimator choice is required. In this section, we provide our rationale for selecting the tail estimator from [MMO15] by highlighting its strengths.
The main arguments are as follows: (i) the estimator's theoretical framework is established through its convergence in distribution to the true tail index (via a Central Limit Theorem result, detailed in Theorem 2.3 [MMO15]), which is further complemented by its demonstrated asymptotic consistency (as per Corollary 2.4 [MMO15]). (ii) The estimator has already been used in various articles and its qualities have been thoroughly examined. For example, this can be observed in Figure 14 (taken from
[SSG19]), which, in our opinion, contains convincing estimation results (e.g., small error bars regardless of the magnitude of the true tail index).
Figure 14: Evaluation of the estimator from [MMO15]. Figure is directly taken from [SSG19].
F Strongly Convex Problem Example
Consider the following one-dimensional logistic regression where the regularization parameter  $\lambda \sim Exp(\mu)$ , i.e., it follows an Exponential distribution with mean  $1 / \mu$  - its probability density function is as follows:
$$
f (\lambda ; \mu) = \left\{ \begin{array}{l l} \mu^ {- 1} e ^ {- \lambda / \mu} & \lambda \geq 0 \\ 0 & \lambda <   0 \end{array} \right..
$$
As before, let  $(a_i,q_i)_{i\geq 1}$  be i.i.d. random variables in  $\mathbb{R}^2$ , such that  $z_{i}\equiv (a_{i},q_{i})$ . In this scenario, where  $a\sim N(0,\sigma^2)$  and  $y\in \{0,1\}$ , the loss function equals:
$$
\ell (x, z) = - y \ln \left(\frac {1}{\mathrm {e} ^ {- a x} + 1}\right) - (1 - y) \ln \left(1 - \frac {1}{\mathrm {e} ^ {- a x} + 1}\right) + \frac {1}{2} \lambda \| x \| ^ {2}.
$$
Now, the second derivative of the loss with respect to the parameter  $x$  equals  $\nabla^2\ell (x,z) = \frac{a^2\mathrm{e}^{ax}}{(\mathrm{e}^{ax} + 1)^2} +\lambda$ . Note that  $|\nabla^2\ell (x,z)|\leq a^2 /4 + \lambda ,\forall x\in X$ . In the SGD case, from [HM21], we have  $r(z) = \lim \inf_{\| x\| \to \infty}\sigma_{\min}\left(\mathrm{I}_d - \gamma \nabla^2\ell (x,z)\right)$  and  $R(z) = \sup_x\left\| \mathrm{I}_d - \gamma \nabla^2\ell (x,z)\right\|$ . In other words, as we are in the one-dimensional setting, we have:
$$
r (z) = \lim  _ {| x | \rightarrow \infty} \inf  \left| 1 - \gamma \lambda - \gamma \frac {a ^ {2} \mathrm {e} ^ {a x}}{(\mathrm {e} ^ {a x} + 1) ^ {2}} \right| = | 1 - \gamma \lambda |, a n d
$$
$$
R (z) = \sup _ {x} \left| 1 - \gamma \lambda - \gamma \frac {a ^ {2} \mathrm {e} ^ {a x}}{(\mathrm {e} ^ {a x} + 1) ^ {2}} \right| = \max \left(| 1 - \gamma \lambda |, | 1 - \gamma \lambda - \gamma \frac {a ^ {2}}{4} |\right).
$$
Now, we require  $\mathbb{P}(r(Z_1) > 1) > 0$  and  $\mathbb{E}[R(Z_1)] < 1$ . As  $\lambda \sim Exp(\mu)$ , we have that  $\mathbb{P}(r(Z_1) > 1) > 0$ . For the latter condition, we can calculate the required expectation:
$$
\begin{array}{l} \mathbb {E} | 1 - \gamma \lambda | = \int_ {0} ^ {\frac {1}{\gamma}} (1 - \gamma \lambda) \mu \mathrm {e} ^ {- \mu \lambda} \mathrm {d} \lambda + \int_ {\frac {1}{\gamma}} ^ {\infty} (\gamma \lambda - 1) \mu \mathrm {e} ^ {- \mu \lambda} \mathrm {d} \lambda \\ = - \frac {(\gamma - \mu) - \gamma \mathrm {e} ^ {- \frac {\mu}{\gamma}}}{\mu} + \frac {\gamma \mathrm {e} ^ {- \frac {\mu}{\gamma}}}{\mu} \\ = \frac {2 \gamma \mathrm {e} ^ {- \frac {\mu}{\gamma}} - \gamma}{\mu} + 1. \\ \end{array}
$$
Furthermore, we have:
$$
\begin{array}{l} \mathbb {E} \left| 1 - \gamma \frac {x ^ {2}}{4} - \gamma \lambda \right| = \mathbb {E} \int_ {0} ^ {1 - \frac {\gamma x ^ {2}}{4}} \left(1 - \gamma \frac {x ^ {2}}{4} - \gamma \lambda\right) \mu \mathrm {e} ^ {- \mu \lambda} \mathrm {d} \lambda + \mathbb {E} \int_ {1 - \frac {\gamma x ^ {2}}{4}} ^ {\infty} \left(\gamma \frac {x ^ {2}}{4} + \gamma \lambda - 1\right) \mu \mathrm {e} ^ {- \mu \lambda} \mathrm {d} \lambda \\ = \mathbb {E} \left[ \frac {2 \gamma}{\mu} \mathrm {e} ^ {\frac {\mu}{\gamma} (1 - \frac {\gamma x ^ {2}}{4})} + 1 - \frac {\gamma}{\mu} - \frac {\gamma x ^ {2}}{4} \right] \\ = \int_ {- \infty} ^ {\infty} \left(\frac {2 \gamma}{\mu} \mathrm {e} ^ {\frac {\mu}{\gamma} (1 - \frac {\gamma x ^ {2}}{4})} + 1 - \frac {\gamma}{\mu} - \frac {\gamma x ^ {2}}{4}\right) \frac {\mathrm {e} ^ {- \frac {1}{2 \sigma^ {2}} x ^ {2}}}{\sqrt {2 \pi \sigma^ {2}}} \mathrm {d} x \\ = \frac {2 \sqrt {2} \gamma}{\mu \sqrt {2 - \sigma^ {2} \mu}} \mathrm {e} ^ {- \frac {\mu}{\gamma}} - \gamma (\frac {1}{\mu} + \frac {\sigma^ {2}}{4}) + 1 \\ \end{array}
$$
We can see that, for example, both expressions are less than 1 in modulus for  $\mu = 0.1$ ,  $\sigma^2 = 1$ , and  $\gamma = 0.1$ . It is important to highlight that the aforementioned calculations necessitate the condition  $\frac{1}{2\sigma^2} - \frac{\mu}{4} > 0$ , or equivalently  $\sigma^2\mu < 2$ . This condition can be interpreted as a stability criterion, indicating that the mean of the penalization term  $\lambda$  must increase as the variance of the data grows. In other words, a larger variance necessitates a higher mean value for  $\lambda$ .