SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Krunoslav Lehman Pvasovic
Inria Paris, CNRS, Ecole Normale Supérieure, PSL Research University
krunoslav.lehman-pavasovic@inria.fr
Alain Durmus
CMAP, CNRS, Ecole Polytechnique, Institut Polytechnique de Paris
alain.durmus@polytechnique.edu
Umut Şimşekli
umut.simsekli@inria.fr
A recent line of empirical studies has demonstrated that SGD might exhibit a heavy-tailed behavior in practical settings, and the heaviness of the tails might correlate with the overall performance. In this paper, we investigate the emergence of such heavy tails. Previous works on this problem only considered, up to our knowledge, online (also called single-pass) SGD, in which the emergence of heavy tails in theoretical findings is contingent upon access to an infinite amount of data. Hence, the underlying mechanism generating the reported heavy-tailed behavior in practical settings, where the amount of training data is finite, is still not well-understood. Our contribution aims to fill this gap. In particular, we show that the stationary distribution of offline (also called multi-pass) SGD exhibits 'approximate' power-law tails and the approximation error is controlled by how fast the empirical distribution of the training data converges to the true underlying data distribution in the Wasserstein metric. Our main takeaway is that, as the number of data points increases, offline SGD will behave increasingly 'power-law-like'. To achieve this result, we first prove nonasymptotic Wasserstein convergence bounds for offline SGD to online SGD as the number of data points increases, which can be interesting on their own. Finally, we illustrate our theory on various experiments conducted on synthetic data and neural networks.
Many machine learning problems can be cast as the following population risk minimization problem:
$$
\operatorname {m i n i m i z e} x \mapsto F (x) := \mathbb {E} [ f (x, Z) ], \quad Z \sim \mu_ {z}, \tag {1}
$$
where  $x \in \mathbb{R}^d$  denotes the model parameters,  $\mu_z$  is the data distribution over the measurable space  $(\mathsf{Z},\mathcal{Z})$  and  $f: \mathbb{R}^d \times \mathsf{Z} \to \mathbb{R}$  is a loss function. The main difficulty in addressing this problem is that  $\mu_z$  is typically unknown. Suppose we have access to an infinite sequence of independent and
identically distributed (i.i.d.) data samples  $\mathcal{D} \coloneqq \{Z_1, Z_2, \ldots\}$  from  $\mu_z$ . In that case, we can resort to the online stochastic gradient descent (SGD) algorithm, which is based on the following recursion:
$$
X _ {k + 1} = X _ {k} - \frac {\eta}{b} \sum_ {i \in \Omega_ {k + 1}} \nabla f \left(X _ {k}, Z _ {i}\right), \tag {2}
$$
where  $k$  denotes the iterations,  $\Omega_k \coloneqq \{b(k - 1) + 1, b(k - 1) + 2, \ldots, bk\}$  is the batch of data-points at iteration  $k$ ,  $\eta$  denotes the step size,  $b$  denotes the batch size such that  $|\Omega_k| = b$ , and  $Z_i$  with  $i \in \Omega_k$ , denotes the  $i$ -th data sample at the  $k$ -th iteration. This algorithm is also called 'single-pass' SGD, as it sees each data point  $Z_i$  in the infinite data sequence  $\mathcal{D}$  only once.
While drawing i.i.d. data samples at each iteration is possible in certain applications, in the majority of practical settings, we only have access to a finite number of data points, preventing online SGD use. More precisely, we have access to a dataset of  $n$  i.i.d. points  $\mathcal{D}_n \coloneqq \{Z_1, \ldots, Z_n\}^1$ , and given these points the goal is then to minimize the empirical risk  $\hat{F}^{(n)}$ , given as follows:
$$
\operatorname {m i n i m i z e} x \mapsto \hat {F} ^ {(n)} (x) := \frac {1}{n} \sum_ {i = 1} ^ {n} f (x, Z _ {i}) \text {o v e r} \mathbb {R} ^ {d}. \tag {3}
$$
To attack this problem, one of the most popular approaches is the offline version of (2), which is based on the following recursion:
$$
X _ {k + 1} ^ {(n)} = X _ {k} ^ {(n)} - \frac {\eta}{b} \sum_ {i \in \Omega_ {k + 1} ^ {(n)}} \nabla f \left(X _ {k} ^ {(n)}, Z _ {i}\right), \tag {4}
$$
where  $\Omega_k^{(n)}\subset \{1,\ldots ,n\}$  denotes the indices of the (uniformly) randomly chosen data points at iteration  $k$  with  $|\Omega_k^{(n)}| = b\leq n$  and  $\cdot^{(n)}$  emphasizes the dependence on the sample size  $n$ . Analogous to the single-pass regime, this approach is also called the multi-pass SGD, as it requires observing the same data points multiple times.
Despite its ubiquitous use in modern machine learning applications, the theoretical properties of offline SGD have not yet been well-established. Among a plethora of analyses addressing this question, one promising approach has been based on the observation that parameters learned by SGD can exhibit heavy tails. In particular,  $\left[\mathrm{SGN}^{+}19\right]$  have empirically demonstrated a heavy-tailed behavior for the sequence of stochastic gradient noise:
$$
\left(\nabla \hat {F} ^ {(n)} (X _ {k} ^ {(n)}) - b ^ {- 1} \sum_ {i \in \Omega_ {k + 1} ^ {(n)}} \nabla f (X _ {k} ^ {(n)}, Z _ {i})\right) _ {k \geq 1}.
$$
Since then, further studies have extended this observation to other sequences appearing in machine learning algorithms [ZFM $^{+}$ 20, MM19, ZLMU22]. These results were recently extended by  $\left[\mathrm{BSE}^{+}21\right]$ , who showed that the parameter sequence  $\left(X_{k}^{(n)}\right)_{k\geq 1}$  itself can also exhibit heavy tails for large  $\eta$  and small  $b$ . These empirical investigations all hinted a connection between the observed heavy tails and the generalization performance of SGD: heavier the tails might indicate a better generalization performance.
Motivated by these empirical findings, several subsequent papers theoretically investigated how heavy-tailed behavior can emerge in stochastic optimization. In this context, [GSZ21] have shown that when online  $SGD$  is used with a quadratic loss (i.e.,  $f(x,z = (a,y)) = 2^{-1}(a^{\top}x - y)^{2}$ ), the distribution of the iterates  $(X_{k})_{k\geq 0}$  can converge to a heavy-tailed distribution, even with exponentially light-tailed data (e.g., Gaussian). More precisely, for a fixed step-size  $\eta >0$ , they showed that, under appropriate assumptions, there exist constants  $c\in \mathbb{R}_+$  and  $\alpha >0$ , such that the following identity holds:
$$
\lim  _ {t \rightarrow \infty} t ^ {\alpha} \mathbb {P} \left(\left\| X _ {\infty} \right\| > t\right) = c, \tag {5}
$$
where  $X_{\infty} \sim \pi$  and  $\pi$  denotes the stationary distribution of online SGD (2). This result illustrates that the distribution of the online SGD iterates follows a power-law decay with the tail index  $\alpha$ :  $\mathbb{P}\left(\| X_{\infty}\| >t\right) \approx t^{-\alpha}$  for large  $t$ . Furthermore,  $\alpha$  depends monotonically on the algorithm
Here we deliberately choose the same notation  $Z_{i}$  for both infinite and finite data regimes to highlight the fact that we can theoretically view the finite data regime from the following perspective: given an infinite sequence of data points  $\mathcal{D}$ , the finite regime only uses the first  $n$  elements of  $\mathcal{D}$ , which constitute  $\mathcal{D}_n$ .
Figure 1: Left: histograms of the parameter norms for Gaussian data linear regression. Right: histograms of NN parameter norms for the first layer, trained on MNIST. Norms exceeding the  $\mu +2\sigma$  threshold are marked with a red asterisk.
hyperparameters  $\eta$ ,  $b$ , and certain properties of the data distribution  $\mu_z$ . This result is also consistent with [DMN+21, Example 1], which shows a simple instance from linear stochastic approximation such that for any fixed step-size  $\eta$ , there exists  $p_c > 0$  such that for any  $p \geq p_c$ ,  $\lim_{k \to +\infty} \mathbb{E}[\|X_k\|^p] = +\infty$ . In a concurrent study, [HM21] showed that this property is, in fact, not specific for quadratic problems and can hold for more general loss functions and different choices of stochastic optimizers, with the constraint that i.i.d. data samples are available at every iteration<sup>2</sup>.
Another line of research has investigated the theoretical links between heavy tails and the generalization performance of SGD. Under different theoretical settings and based on different assumptions, [SSDE20, BSE $^{+}$ 21, LW $\S$ 22, RBG $^{+}$ 22, HSKM22, RZGS $\S$ 23] proved generalization bounds (i.e., bounds on  $|\hat{F}^{(n)}(x) - F(x)|$ ) illustrating that the heavy tails can be indeed beneficial for better performance. However, all these results rely on exact heavy tails, which, to our current knowledge, can only occur in the online SGD regime where there is access to an infinite sequence of data points. Hence the current theory (both on the emergence of heavy tails and their links to generalization performance) still falls short in terms of explaining the empirical heavy-tailed behavior observed in offline SGD as reported in [SGN $^{+}$ 19, MM19, ZFM $^{+}$ 20, BSE $^{+}$ 21].
Main problematic and contributions. Although empirically observed, it is currently unknown how heavy-tailed behavior arises in offline SGD since the aforementioned theory [GSZ21, HM21] requires infinite data. Our main goal in this paper is hence to develop a theoretical framework to catch how and in what form heavy tails may arise in offline SGD, where the dataset is finite.
We build our theory based on two main observations. The first observation is that, since we have finitely many data points in offline SGD, the moments (of any order) of the iterates  $X_{k}^{(n)}$  can be bounded (see, e.g., [CGZ19]), hence in this case we cannot expect exact power-law tails in the form of (5) in general. Our second observation, on the other hand, is that since the finite dataset  $\mathcal{D}_n$  can be seen as the first  $n$  elements of the infinite sequence  $\mathcal{D}$ , as  $n$  increases the offline SGD recursion (4) should converge to the online SGD recursion (2) in some sense. Hence, when online SGD shows a heavy-tailed behavior (i.e., the case where  $n \to +\infty$ ), as we increase  $n$ , we can expect that the tail behavior of offline SGD should be more and more 'power-law-like'.
To further illustrate this observation, as a preliminary exploration, we run offline SGD in a 100-dimensional linear regression problem, as well as a classification problem on the MNIST dataset, using a fully-connected, 3-layer neural network.
Figure 2: Estimated tail indices
In Figure 1, we plot the histograms of the parameter norms after 1000 (and 5000
resp.) iterations. We observe the following consistent patterns:  $(i)$  the means and standard deviations of the parameters' estimated stabilizing distributions increase with  $n$ , and  $(ii)$ , the quantity and magnitude of the parameters far from the bulk of the distribution increase with  $n$ . In other words, as we increase the number of samples  $n$  in offline SGD, the behavior of the iterates becomes heavier-tailed.
2We will present these results in more detail in Section 2.
3The exact experimental details can be found in Section 4.
As another example, we utilize the tail-estimator from  $\left[\mathrm{GSZ21}\right]^4$  to compare the estimated tail indices  $\alpha$  for offline and online SGD. The results are plotted in Figure 2 and correspond to 10 random initializations with different step-sizes (for further details, see Tail estimation). We observe that, as  $n$  increases, the estimated tail indices for offline SGD (shown in green) get closer to the true tail index corresponding to online SGD (the horizontal blue line).
Our main contribution in this paper is to make these observations rigorous and explicit. In particular, we extend the tail estimation results for online SGD [GSZ21, HM21] to offline SGD, and show that the stationary distribution of offline SGD iterates will exhibit 'approximate' heavy tails. Informally, for both quadratic and a class of strongly convex losses, we show that, with high probability, there exist constants  $c_{1}, c_{2}$ , such that for large enough  $t$  we have the following tail bound:
$$
\left[ \frac {c _ {1}}{t ^ {\alpha}} - \frac {1}{t} \mathbf {W} _ {1} \left(\mu_ {z}, \mu_ {z} ^ {(n)}\right) \right] \lesssim \mathbb {P} \left(\| X _ {\infty} ^ {(n)} \| > t\right) \lesssim \left[ \frac {c _ {2}}{t ^ {\alpha}} + \frac {1}{t} \mathbf {W} _ {1} \left(\mu_ {z}, \mu_ {z} ^ {(n)}\right) \right], \tag {6}
$$
where  $\alpha$  is the tail index of online SGD as given in (5),  $X_{\infty}^{(n)} \sim \pi^{(n)}$  denotes a sample from the stationary distribution  $\pi^{(n)}$  of offline SGD (4),  $\mu_z^{(n)} = n^{-1}\sum_{i=1}^n \delta_{Z_i}$  denotes the empirical measure of the data points in the data sample  $\mathcal{D}_n = \{Z_1, \ldots, Z_n\}$ , and  $\mathbf{W}_1$  denotes the Wasserstein-1 distance (to be defined formally in the next section).
Our result indicates that the tail behavior of offline SGD is mainly driven by two terms: (i) a persistent term  $(c / t^{\alpha})$  that determines the power-law decay, and (ii) a vanishing term  $\mathbf{W}_1(\mu_z,\mu_z^{(n)}) / t$ , as we take  $n\to \infty$ . In other words, compared to (5), we can see that, with high-probability,  $X_{\infty}^{(n)}$  exhibits an approximate power-law decay behavior, with a discrepancy controlled by  $\mathbf{W}_1(\mu_z,\mu_z^{(n)})$ .
Fortunately, the term  $\mathbf{W}_1(\mu_z,\mu_z^{(n)})$  has been well-studied in the literature. By combining our results with Wasserstein convergence theorems for empirical measures [FG15], we further present nonasymptotic tail bounds for offline SGD. Our main takeaway is as follows: as  $n$  increases, offline SGD will exhibit approximate power-laws, where the approximation error vanishes with a rate given by how fast the empirical distribution of the data points  $\mu_z^{(n)}$  converges to the true data distribution  $\mu_z$  as  $n\to \infty$ .
To prove our tail bounds, as an intermediate step, we prove nonasymptotic Wasserstein convergence results for offline SGD in the form:  $\mathbf{W}_1(\pi ,\pi^{(n)})\lesssim \mathbf{W}_1(\mu_z,\mu_z^{(n)})$ , with high probability, recalling that  $\pi$  and  $\pi^{(n)}$  respectively denote online and offline SGD stationary distributions. These results, we believe, hold independent interest for the community. Finally, we support our theory with various experiments conducted on both synthetic quadratic optimization problems and real neural network settings.
2 Preliminaries and Technical Background
2.1 Notation and preliminary definitions
We let  $\mathrm{I}_d$  denote the  $d\times d$  identity matrix. We denote  $[n] = \{1,\dots ,n\}$ ,  $\mathcal{P}_{n,b} = \{\mathsf{l}\subset [n],|\mathsf{l}| = b\}$ . For a vector  $x\in \mathbb{R}^d$ ,  $\| x\|$  denotes the Euclidean norm of  $x$ , and for matrix  $\mathbf{A}$ , the norm  $\| \mathbf{A}\| = \sup_{\| x\| = 1}\| \mathbf{A}x\|$  denote its spectral norm. We denote with  $\sigma_{\mathrm{min}}(\mathbf{A})$ ,  $\sigma_{\mathrm{max}}(\mathbf{A})$  the smallest and largest singular values of  $\mathbf{A}$ , respectively.  $\mathcal{P}\left(\mathbb{R}^d\right)$  is the set of all probability measures on  $\mathbb{R}^d$ , and  $\mathcal{L}(X)$  denotes the probability law of a random variable  $X$ . We denote the set of all couplings of two measures  $\mu$  and  $\nu$  with  $\Gamma (\mu ,\nu)$ . Finally, for two functions  $f(n)$  and  $g(n)$  defined on  $\mathbb{R}_+$ , we denote  $f = \mathcal{O}(g)$ , if there exist constants  $c\in \mathbb{R}_+$ ,  $n_0\in \mathbb{N}$ , such that  $f(n)\leq cg(n),\forall n > n_0$ . The Dirac measure concentrated at  $x$  is defined as follows: for a measurable set  $E$ ,  $\delta_x(E) = 1$  if  $x\in E$ ,  $\delta_x(E) = 0$ , otherwise.
For  $p \in [1, \infty)$ , the Wasserstein- $p$  distance between two distributions  $\mu$  and  $\nu$  on  $\mathbb{R}^d$  is defined as:
$$
\mathbf {W} _ {p} (\mu , \nu) := \inf  _ {\gamma \in \Gamma (\mu , \nu)} \left(\int \| x - y \| ^ {p} d \gamma (x, y)\right) ^ {1 / p}.
$$
We correspondingly define the Wasserstein-  $p$  distance between two distributions  $\mu$  and  $\nu$  on  $\mathbb{R}^{d\times d}$  as
$$
\underline {{\mathbf {W} _ {p} (\mu , \nu)}} := \inf  _ {\gamma \in \Gamma (\mu , \nu)} \left(\int \| \mathbf {A} - \mathbf {B} \| ^ {p}   \mathrm {d} \gamma (\mathbf {A}, \mathbf {B})\right) ^ {1 / p}.
$$
4The justification for the estimator choice can be found in the appendix, Sec. E.
2.2 Heavy tails in online SGD
We first formally define heavy-tailed distributions with a power-law decay, used throughout our work.
Definition 1 (Heavy-tailed distributions with a power-law decay). A random vector  $X$  is said to follow a distribution with a power-law decay if  $\lim_{t\to \infty}t^{\alpha}\mathbb{P}(\| X\| \geq t) = c_0$ , for constants  $c_{0} > 0$  and  $\alpha >0$ , the latter known as the tail index.
The tail index  $\alpha$  determines the thickness of the tails: larger values of  $\alpha$  imply a lighter tail.
Recently, [GSZ21] showed that online SGD might produce heavy tails even with exponentially light-tailed data. More precisely, they focused on a quadratic optimization problem, where  $z \equiv (a,q) \in \mathbb{R}^{d + 1}$  and  $f(x,z) = 2^{-1}(a^{\top}x - q)^{2}$ . With this choice, the online SGD recursion reads as follows:
$$
X _ {k + 1} = X _ {k} - \frac {\eta}{b} \sum_ {i \in \Omega_ {k + 1}} \left(a _ {i} a _ {i} ^ {\top} X _ {k} - a _ {i} q _ {i}\right) = \left(\mathrm {I} _ {d} - \eta \mathbf {A} _ {k + 1}\right) X _ {k} + \eta \mathbf {b} _ {k + 1} \tag {7}
$$
where  $(Z_{k})_{k\geq 1}\equiv (a_{k},q_{k})_{k\geq 1}$ $\mathbf{A}_k\coloneqq b^{-1}\sum_{i\in \Omega_k}a_ia_i^\top$  , and  $\mathbf{b}_k\coloneqq b^{-1}\sum_{i\in \Omega_k}a_ia_i$ .
In this setting, they proved the following theorem.
Theorem 1 ([GSZ21]). Assume that  $(a_{k},q_{k})_{k\geq 1}$  are i.i.d. random variables such that  $a_1\sim$ $\mathrm{N}(0,\sigma^2\mathrm{I}_d)$ , with  $\sigma^2 >0$  and  $q_{i}$  has a continuous density with respect to the Lebesgue measure on  $\mathbb{R}$  with all its moments being finite. In addition suppose that  $\mathbb{E}[\log \| \mathrm{I}_d - \eta \mathbf{A}_1\| ] < 0$ . Then, there exists a unique  $\alpha >0$  such that  $\mathbb{E}[\| \mathrm{I}_d - \eta \mathbf{A}_1\|^\alpha ] = 1$  and the iterates  $(X_{k})_{k\geq 0}$  in (7) converge in distribution to a unique stationary distribution  $\pi$  such that if  $X_{\infty}\sim \pi$  and  $\alpha \notin \mathbb{N}$ :
$$
\lim  _ {t \rightarrow \infty} t ^ {\alpha} \mathbb {P} \left(\left\| X _ {\infty} \right\| > t\right) = c, \quad \text {w i t h} c \in \mathbb {R} _ {+}. \tag {8}
$$
The authors of [GSZ21] also showed that the tail index  $\alpha$  is monotonic with respect to the algorithm hyperparameters  $\eta$  and  $b$ : a larger  $\eta$  or smaller  $b$  indicates heavier tails, i.e., smaller  $\alpha^5$ .
In a concurrent study, [HM21] considered a more general class of loss functions and choices of stochastic optimization algorithms. They proved a general result, where we translate it for online SGD applied on strongly convex losses in the following theorem.
Theorem 2 ([HM21]). Assume that for every  $z \in \mathbb{Z}$ ,  $f(\cdot, z)$  is twice differentiable and is strongly convex. Consider the recursion in (2) with a sequence of i.i.d. random variables  $(Z_k)_{k \geq 1}$  and let  $R$  and  $r$  be two non-negative functions defined as: for any  $z \in \mathbb{Z}$ ,
$$
R (z) := \sup  _ {x \in \mathbb {R} ^ {d}} \left\| \mathrm {I} _ {d} - \eta \nabla^ {2} f (x, z) \right\|, a n d r (z) := \lim  \inf  _ {\| x \| \to \infty} \sigma_ {\min } \left(\mathrm {I} _ {d} - \eta \nabla^ {2} f (x, z)\right).
$$
Further assume that  $\mathbb{E}[R(Z_1) + \| \nabla f(x^\star, Z_1)\|] < +\infty$ , for some  $x^\star \in \mathbb{R}^d$ ,  $\mathbb{E}[\log R(Z_1)] < 0$ , and  $\mathbb{P}(r(Z_1) > 1) > 0$ . Then, the iterates  $(X_k)_{k \geq 1}$  in (2) admit a stationary distribution  $\pi$ . Moreover, there exist  $\alpha, \beta > 0$  such that  $\mathbb{E}[r(Z)^\alpha] = 1$ ,  $\mathbb{E}[R(Z)^\beta] = 1$ , and for any  $\varepsilon > 0$ :
$$
\lim  _ {t \to \infty} t ^ {\alpha + \varepsilon} \mathbb {P} \left(\| X _ {\infty} \| > t\right) > 0, \quad a n d \quad \lim  _ {t \to \infty} t ^ {\beta - \varepsilon} \mathbb {P} \left(\| X _ {\infty} \| > t\right) <   + \infty ,
$$
where  $X_{\infty}\sim \pi$
The result illustrates that power laws can arise in general convex stochastic optimization algorithms. We note that in general, Theorem 2 does not require each  $f(\cdot, z)$  to be strongly convex, in fact, it can accommodate non-convex functions as well. However, we are not aware of any popular non-convex machine learning problem that can satisfy all the assumptions of Theorem 2. Nevertheless, for better illustration, in Section F, we show that an  $\ell_2$ -regularized logistic regression problem with random regularization coefficients falls into the scope of Theorem 2.
The two aforementioned theorems are limited to the online setting, and it is unclear whether they apply to a finite number of data points  $n$ . In the subsequent section, we aim to extend these theorems to the offline setting.
Without the Gaussian data assumption, [GSZ21] proved lower-bounds over  $\alpha$  instead of the exact identification of  $\alpha$  as given in (8). We also note that the condition  $\alpha \notin \mathbb{N}$  is not required in [GSZ21]: without such a condition, one can show that, for any  $u\in \mathbb{R}^d$  with  $\| u\| = 1$ ,  $\lim_{t\to \infty}t^{\alpha}\mathbb{P}\left(u^{\top}X_{\infty} > t\right)$  has a non-trivial limit. For the clarity of the presentation, we focus on the tails of the norm  $\| X_{\infty}\|$ , hence use a non-integer  $\alpha$ , which leads to (8) (for the equivalence of these expressions, see  $[\mathrm{BDM}^{+}16$ , Theorem C.2.1]).
3.1 Quadratic objectives
We first focus on the quadratic optimization setting considered in Theorem 1 as its proof is simpler and more instructive. Similarly as before, let  $(a_i,q_i)_{i\geq 1}$  be i.i.d. random variables in  $\mathbb{R}^{d + 1}$ , such that  $z_{i}\equiv (a_{i},q_{i})$ . Following the notation from (4), we can correspondingly define the offline SGD recursion as follows:
$$
X _ {k + 1} ^ {(n)} = X _ {k} ^ {(n)} - \frac {\eta}{b} \sum_ {i \in \Omega_ {k + 1} ^ {(n)}} \left(a _ {i} a _ {i} ^ {\top} X _ {k} ^ {(n)} - a _ {i} q _ {i}\right) = \left(\mathrm {I} _ {d} - \eta \mathbf {A} _ {k + 1} ^ {(n)}\right) X _ {k} ^ {(n)} + \eta \mathbf {b} _ {k + 1} ^ {(n)}, \tag {9}
$$
where  $\mathbf{A}_k^{(n)}\coloneqq b^{-1}\sum_{i\in \Omega_k^{(n)}}a_ia_i^\top$ ,  $\mathbf{b}_k^{(n)}\coloneqq b^{-1}\sum_{i\in \Omega_k^{(n)}}a_iq_i$ , and  $\Omega_k^{(n)}$  is as defined in (4). Now,  $(\mathbf{A}_k^{(n)},\mathbf{b}_k^{(n)})_{k\geq 1}$  are i.i.d. random variables with respect to the empirical measure:
$$
\mu_ {\mathbf {A}, \mathbf {b}} ^ {(n)} = \binom {n} {b} ^ {- 1} \sum_ {i = 1} ^ {\binom {n} {b}} \delta_ {\left\{\overline {{\mathbf {A}}} _ {i} ^ {(n)}, \overline {{\mathbf {b}}} _ {i} ^ {(n)} \right\}}, \tag {10}
$$
where we enumerate all possible choices of minibatch indices  $\mathcal{P}_{n,b} = \{\mathsf{l}\subset \{1,\ldots ,n\} :|\mathsf{l}| = b\}$  as  $\mathcal{P}_{n,b} = \{\mathsf{S}_1^{(n)},\mathsf{S}_2^{(n)},\dots ,\mathsf{S}_{\binom{n}{b}}^{(n)}\}$  and  $(\overline{\mathbf{A}}_i^{(n)},\overline{\mathbf{b}}_i^{(n)})_{i = 1}^{\binom{n}{b}}$  are i.i.d. random variables defined as follows:
$$
\overline {{\mathbf {A}}} _ {i} ^ {(n)} = \frac {1}{b} \sum_ {j \in \mathbb {S} _ {i} ^ {(n)}} a _ {j} a _ {j} ^ {\top}, \quad \overline {{\mathbf {b}}} _ {i} ^ {(n)} = \frac {1}{b} \sum_ {j \in \mathbb {S} _ {i} ^ {(n)}} a _ {j} q _ {j}. \tag {11}
$$
We denote by  $\mu_{\mathbf{A},\mathbf{b}}$  the common distribution of  $(\overline{\mathbf{A}}_i^{(n)},\overline{\mathbf{b}}_i^{(n)})_{i = 1}^{(n)}$ . Note that it does not depend on  $n$  but only  $b$ .