SlowGuess/NeurIPS_2023_OCR huggingface dataset:
With these definitions at hand, we define the two marginal measures of  $\mu_{\mathbf{A},\mathbf{b}}$  and  $\mu_{\mathbf{A},\mathbf{b}}^{(n)}$  with  $\mu_{\mathbf{A}}$ ,  $\mu_{\mathbf{b}}$ , and  $\mu_{\mathbf{A}}^{(n)}, \mu_{\mathbf{b}}^{(n)}$ , respectively. Before proceeding to the theorem, we have to define Wasserstein ergodicity. A discrete-time Markov chain  $(Y_k)_{k \geq 0}$  is said to be (Wasserstein-1) geometrically ergodic if it admits a stationary distribution  $\pi_Y$  and there exist constants  $c \geq 0$  and  $\rho \in (0,1)$ , such that
$$
\mathbf {W} _ {1} \left(\mathcal {L} \left(Y _ {k}\right), \pi_ {Y}\right) \leq c \mathrm {e} ^ {- \rho k}, \text {f o r a n y} k \geq 0. \tag {12}
$$
The chain is simply called (Wasserstein-1) ergodic if  $\lim_{k\to \infty}\mathbf{W}_1(\mathcal{L}(Y_k),\pi_Y) = 0$
We can now state our first main contribution, an extension of Theorem 1 to the offline setting:
Theorem 3. Let  $n \geq 1$  and  $\epsilon_n \in [0,1]$  be the probability that  $(X_k^{(n)})_{k \geq 0}$  is not ergodic (in the Wasserstein sense)<sup>6</sup> with a stationary distribution  $\pi^{(n)}$  having finite q-th moment with  $q > 1$ . Assume that the conditions of Theorem 1 hold with  $\alpha > 1$ . Then, for any  $\epsilon > 0$ , there exist constants  $\tilde{c}_1, \tilde{c}_2$ , and  $t_0 > 0$  such that for all  $\zeta \in (0,1]$ ,  $t > t_0$ , with probability larger than  $1 - \epsilon_n - \zeta$ , the following inequalities hold:
$$
\left[ \frac {1}{2 ^ {\alpha}} \frac {c - \epsilon}{t ^ {\alpha}} - \frac {\tilde {c} _ {1}}{t n ^ {1 / 2}} \sqrt {\log \frac {\tilde {c} _ {2}}{\zeta}} \right] \leq \mathbb {P} \left(\| X _ {\infty} ^ {(n)} \| > t\right) \leq \left[ 2 ^ {\alpha} \frac {c + \epsilon}{t ^ {\alpha}} + \frac {2 \tilde {c} _ {1}}{t n ^ {1 / 2}} \sqrt {\log \frac {\tilde {c} _ {2}}{\zeta}} \right], \tag {13}
$$
where  $c$  is given in (8).
This result shows that whenever the online SGD recursion admits heavy tails, the offline SGD recursion will exhibit 'approximate' heavy tails. More precisely, as can be seen in (13), the tails will have a global power-law behavior due to the term  $t^{-\alpha}$ , where  $\alpha$  is the tail index determined by online SGD. On the other hand, the power-law behavior is only approximate, as we have an additional term, which vanishes at a rate  $\mathcal{O}(n^{-1/2})$ . Hence, as  $n$  increases, the power-law behavior will be more prominent, which might bring an explanation for why heavy tails are still observed even when  $n$  is finite.
To establish Theorem 3, as an intermediate step, we show that  $\pi^{(n)}$  converges to  $\pi$  in the Wasserstein-1 metric. The result is given in the following theorem and can be interesting on its own.
<sup>6</sup>Since we are assuming Gaussian data,  $\epsilon_{n}$  will converge to zero with an exponential rate. However, we do not focus on explicit calculations of this quantity as it is rather orthogonal to our problematic.
Theorem 4. Under the setting of Theorem 3, for  $p, q > 0$  with  $1/p + 1/q = 1$ ,  $p > d/2$ ,  $q < \alpha$ , the following holds with probability larger than  $1 - \epsilon_n$ :
$$
\mathbf {W} _ {1} \left(\pi , \pi^ {(n)}\right) \leq c _ {0} \frac {\eta}{1 - \delta_ {A}} \mathbf {W} _ {p} \left(\mu_ {\mathbf {A}, \mathbf {b}}, \mu_ {\mathbf {A}, \mathbf {b}} ^ {(n)}\right), \tag {14}
$$
where  $c_{0} = (\mathbb{E}[||X_{0}^{(n)}||^{q}])^{1 / q} + 1$  with  $X_0^{(n)}\sim \pi^{(n)}$ , and  $\delta_A = \mathbb{E}[\| \mathbf{I}_d - \eta \mathbf{A}_1\| ]$ .
This result shows that the stationary distribution of offline SGD will converge to the one of online SGD as  $n\to \infty$  , and the rate of this convergence is determined by how fast the empirical distribution of the dataset converges to the true data distribution, as measured by  $\mathbf{W}_p(\mu_{\mathbf{A},\mathbf{b}},\mu_{\mathbf{A},\mathbf{b}}^{(n)})$  . Once (14) is established, to obtain Theorem 3, we use the relationship that  $\mathbf{W}_1(\pi ,\pi^{(n)})\geq$ $\mathbf{W}_1(\mathcal{L}(\| X_\infty^{(n)}\|),\mathcal{L}(\| X_\infty \|)) = \int_0^\infty |\mathbb{P}(\| X_\infty^{(n)}\| >t) - \mathbb{P}(\| X_\infty \| >t)|  dt$  (see Lemmas 1 and 3 in the Appendix). Finally, by exploiting the assumptions made on  $\mu_{\mathbf{A},\mathbf{b}}$  in Theorem 1, we can invoke [FG15] (see Lemma 2 in Section A.1) and show that  $\mathbf{W}_p(\mu_{\mathbf{A},\mathbf{b}},\mu_{\mathbf{A},\mathbf{b}}^{(n)})$  behaves as  $\mathcal{O}(n^{-1 / 2})$
3.2 Strongly convex objectives
We now focus on more complex loss functions, as in Theorem 2. For simplicity, we consider  $b = 1$ . The general case, while similar, would complicate the notation and, in our opinion, make our results more difficult to digest. We start by rewriting offline SGD with respect to an empirical measure  $\mu_z^{(n)}$ :
$$
X _ {k + 1} ^ {(n)} = X _ {k} ^ {(n)} - \eta \nabla f \left(X _ {k} ^ {(n)}, \hat {Z} _ {k + 1} ^ {(n)}\right), \tag {15}
$$
where  $(\hat{Z}_k^{(n)})_{k\geq 1}$  are i.i.d. random variables associated with the empirical measure defined for any  $\mathsf{A}\in \mathcal{Z}$  as:
$$
\mu_ {z} ^ {(n)} (\mathsf {A}) = n ^ {- 1} \sum_ {j = 1} ^ {n} \delta_ {\hat {Z} _ {j} ^ {(n)}} (\mathsf {A}). \tag {16}
$$
We first make the following assumption.
Assumption 1. (a) There exists  $L > 0$  such that  $\| \nabla f(x,z) - \nabla f(x,z')\| \leq L(\| x\| +1)(\| z - z'\|),$  for any  $z,z^{\prime}\in \mathbb{Z}$  and  $x\in \mathbb{R}^d$ .
(b) The data distribution  $\mu_z$  has finite  $q$ -th moments with some  $q \geq 1$ .
Assumption 1-(a) is a Lipschitz-like condition that is useful for decoupling  $x$  and  $z$ , and is commonly used in the analysis of optimization and stochastic approximations algorithms; see, e.g., [BWMP12]. We can now state our second main contribution, an extension of Theorem 2 to the offline setting.
Theorem 5. Let  $(X_{k})_{k\geq 0}$  and  $(X_{k}^{(n)})_{k\geq 0}$  be defined as in (2) and (4) with  $b = 1$  and  $n\geq 1$ . Assume Assumption 1 and the conditions of Theorem 2 with  $\beta >1$  hold. Further assume that  $R(Z_{1})$  is non-deterministic (see Theorem 2),  $(X_{k})_{k\geq 0}$  is geometrically ergodic and  $\epsilon_{n}\in [0,1]$  is the probability that  $(X_{k}^{(n)})_{k\geq 0}$  is not ergodic (in the Wasserstein sense). Then, for every  $\varepsilon >0$ , there exist constants  $c,c_{\alpha},c_{\beta}$ , and  $t_0\geq 0$ , such that for all  $t > t_0$ , the following inequalities hold with probability larger than  $1 - \epsilon_{n}$ :
$$
\left[ \frac {1}{2 ^ {\alpha + \epsilon}} \frac {c _ {\alpha}}{t ^ {\alpha + \epsilon}} - \frac {c}{t} \mathbf {W} _ {1} (\mu_ {z}, \mu_ {z} ^ {(n)}) \right] \leq \mathbb {P} (\| X _ {\infty} ^ {(n)} \| > t) \leq \left[ 2 ^ {\beta - \epsilon} \frac {c _ {\beta}}{t ^ {\beta - \epsilon}} + \frac {2 c}{t} \mathbf {W} _ {1} (\mu_ {z}, \mu_ {z} ^ {(n)}) \right], \quad \forall n,
$$
where  $c = (\mathbb{E}[\| X_0^{(n)}\| ] + 1)L\eta /(1 - \delta_R)$  with  $\delta_R = \mathbb{E}[R(Z_1)]$ , and  $X_0^{(n)}\sim \pi^{(n)}$ .
This theorem shows that the approximate power-laws will also be apparent for more general loss functions: for large enough  $n$ , the tails of offline SGD will be upper- and lower-bounded by two power-laws with exponents  $\alpha$  and  $\beta$ . Note that the ergodicity of  $(X_{k})_{k\geq 0}$  can be established by using similar tools provided in [DDB20]. On the other hand, the assumption on  $R(Z_{1})$  being nondeterministic and  $\mathbb{E}[R^{\beta}(Z_1)] = 1$  with  $\beta > 1$  jointly indicate that  $\delta_R = \mathbb{E}[R(Z_1)] < 1$ , which is a form of contraction on average, hence the strong convexity condition is needed.
Similarly to Theorem 3, in order to prove this result, we first obtain a nonasymptotic Wasserstein convergence bound between  $\pi^{(n)}$  and  $\pi$ , stated as follows:
Figure 3: QQ-plots of 1D linear regression experiment. Left: Online SGD exhibits heavy-tails. Right: Offline SGD with varying  $n$ ; larger  $n$  exhibits heavier tails.
Theorem 6. Under the setting of Theorem 5, the following holds with probability greater than  $1 - \epsilon_{n}$ :
$$
\mathbf {W} _ {1} (\pi , \pi^ {(n)}) \leq c _ {0} L \frac {\eta}{1 - \delta_ {R}} \mathbf {W} _ {1} (\mu_ {z}, \mu_ {z} ^ {(n)}),
$$
with  $c_{0} = \mathbb{E}[\| X_{0}^{(n)}\| ] + 1$  where  $X_0^{(n)}\sim \pi^{(n)}$
The convergence rate implied by Theorem 6 is implicitly related to the finite moments of the data-generating distribution  $\mu_z$ . Using the results of [FG15] (Lemma 2 in the Appendix), we can determine the convergence rate  $r$ , where with high probability it will hold that  $\mathbf{W}_1(\pi, \pi^{(n)}) = \mathcal{O}(n^{-r})$ , with  $r \in (0, 1/2]$ . Let the data have finite moments up to order  $q$ . Then, for example, if  $q > 4d$ , we get  $r = 1/2$ , i.e., the fastest rate. As another example: if  $d > 2$  and  $q = 3$ , we obtain a rate  $r = 1/d$ . In other words, the moments of the data-generating distribution determine whether we can achieve a dimension-free rate or fall victim to the curse of dimensionality. Thus, if the data do not admit higher-order moments, our results show that we need a large number of data points  $n$  to be able to observe power-law tails in offline SGD.
In this section, we support our theoretical findings on both synthetic and real-world problems. Our primary objective is to validate the tail behavior of offline SGD iterates by adapting previous online SGD analyses [GSZ21, SSG19], in which the authors investigated how heavy-tailed behavior relates to the step size  $\eta$  and batch size  $b$ , or their ratio  $\eta / b^7$ .
4.1 Linear regression
Experimental setup. In the linear regression examples, we examine the case with a quadratic loss and Gaussian data. In this simple scenario, it was observed that online SGD could exhibit heavily-tailed behavior [GSZ21].
The model can be sum
Figure 4: Estimated tail indices difference
marized as follows: with initial parameters  $\bar{X}_0\sim \mathrm{N}\left(0,\sigma_x^2\mathrm{I}_d\right)$ , features  $a_{i}\sim \mathrm{N}\left(0,\sigma^{2}\mathrm{I}_{d}\right)$ , and targets  $y_{i}\mid a_{i},\bar{X}_{0}\sim \mathrm{N}\left(a_{i}^{\top}\bar{X}_{0},\sigma_{y}^{2}\right)$ , for  $i = 1,\dots ,n$ , and  $\sigma ,\sigma_{x},\sigma_{y} > 0$ . In our experiments, we fix  $\sigma = 1,\sigma_{x} = \sigma_{y} = 3$ , use either  $d = 1$  or  $d = 100$ , and simulate the statistical model to obtain  $\{a_i,y_i\}_{i = 1}^n$ .
In order to examine the offline version of the model (i.e., for offline SGD), we utilize a finite dataset with  $n$  points rather than observing new samples at each iteration. We then analyze the same experimental setting with an increasing number of samples  $n$  and illustrate how similar heavy-tailed behavior emerges in offline SGD.
The code scripts for reproducing the experimental results can be accessed at github.com/krunolp/offline_ht.
Figure 5: Estimated tail indices for a 3-layer fully connected NN
Preliminary tail inspection. We begin by analyzing how the tail behavior differs between online and offline SGD (with varying  $n$  in the latter). Specifically, in Figure 3, we depict a 1-dimensional linear regression task by analyzing the QQ-plots of the estimated stabilizing distributions after 1000 iterations. We observe that online SGD with a sufficiently large  $\eta$  exhibits heavy, non-Gaussian tails, underscoring the prevalence of heavy tails even in such simple scenarios. Furthermore, we also observe that offline SGD exhibits increasingly heavier tails as the sample size  $n$  increases, as our theoretical results suggest.
Tail estimation. We now set  $d = 100$  and run the corresponding offline and online SGD recursions. We then use a tail-index estimator [MMO15], which assumes that the recursions both converge to an exact heavy-tailed distribution. While this is true for online SGD due to Theorem 1, offline SGD will only possess approximate heavy-tails, and the power-law behavior might not be apparent for small  $n$ . Hence, for small  $n$ , we expect the estimated tail index for offline SGD will be inaccurate and get more and more accurate as we increase  $n$ .
We illustrate this in Figure 2, in which we plot the range of estimated offline SGD tail indices (marked in green) corresponding to 10 random initializations, while varying  $n$  and  $\eta$ . We can see that, across all learning rates, the variance of the tail estimation decreases as  $n$  increases and that the estimated values get closer to the estimated tail index for online SGD (marked as the horizontal blue line).
Further analyses. We now run online and offline SGD recursions, varying  $\eta$  from 0.001 to 0.01,  $b$  from 1 to 20, and  $n$  from 1 to 500. Each hyperparameter configuration is run 1600 times with distinct initializations, yielding 1600 estimated samples from the stationary distributions, used to estimate the tail indexes. In order to estimate the tail-indexes  $\alpha$  (online SGD) and  $\alpha^{(n)}$  (offline SGD) of the respective stationary distributions, we follow the procedure as explained in [GSZ21]. Finally, in Figure 4, we plot the absolute difference of the estimated indexes,  $|\hat{\alpha}^{(n)} - \hat{\alpha}|$ .
We find that a larger number of data samples leads to a smaller discrepancy between the online and offline approximations across all batch sizes  $b$  and step sizes  $\eta$ . This trend is consistently observed. Moreover, we observe that larger values of  $\eta$  lead to a smaller discrepancy on average, across all  $n$ . The conclusion here is that, as  $n$  increases, the power-law tails in offline SGD become more apparent and the estimator can identify the true tail index corresponding to online SGD even for moderately large  $n$ , which confirms our initial expectations.
4.2 Neural networks (NN)
To test the applicability of our theory in more practical scenarios, we conduct a second experiment using fully connected (FC) NNs (3 layers, 128 neurons, ReLU activations), as well as larger architectures, such as LeNet (60k parameters, 3 convolutional layers, 2 FC layers)[LBBH98], and AlexNet (62.3M parameters, 5 convolutional layers, 3 FC layers)[KSH17]. The models are trained for 10,000 iterations using cross-entropy loss on the MNIST and CIFAR-10 datasets. We vary the learning rate from  $10^{-4}$  to  $10^{-1}$ , and the batch size  $b$  from 1 to 10, with offline SGD utilizing  $25\%$ ,  $50\%$ , and  $75\%$  of the training data.
We proceed similarly to the linear regression experiment and again replicate the method presented in [GSZ21]: we estimate the tail index per layer, and plot the corresponding results, with different colors representing different data proportions (1.00 indicates the full data set). The results for the fully connected network are presented in Figure 5 (on CIFAR-10 & MNIST), LeNet (on MNIST)
For completeness, LeNet further uses 2 subsampling layers.
Figure 7: Estimated tail indices, LeNet, MNIST
in Figure 7, and AlexNet (on CIFAR-10) in Figure 6. The LeNet CIFAR-10 and AlexNet MNIST results can be found in the Appendix D.
Our observations show that the estimated  $\hat{\alpha}^{(n)}$  has a strong correlation with  $\hat{\alpha}$ : using a reasonably large proportion of the data enables us to estimate the tail index that is measured over the whole dataset. Moreover, although the dependence of  $\hat{\alpha}^{(n)}$  on  $\eta / b$  varies between layers, the
Figure 6: Estimated tail indices, AlexNet, CIFAR-10
measured  $\hat{\alpha}^{(n)}$ 's correlate well with the ratio  $\eta / b$  across all datasets and NN architectures<sup>9</sup>. While our theory does not directly cover this setup, our results show that similar behavior is also observed in more complicated scenarios.
We established a relationship between the data-generating distributions and stationary distributions of offline and online SGD. This enabled us to develop the first theoretical result illuminating the heavy-tailed behavior in offline SGD. We extended previous results encompassing both quadratic losses, as well as more sophisticated strongly convex loss functions. Through an experimental study, we validated our theoretical findings in a variety of settings.
Limitations. There are two main limitations to our work. Firstly, our analysis focuses predominantly on the 'upper' tails of the distributions. Studying the bulk of the distribution of the iterates may reveal new findings. For example, research in this direction could connect our findings to learning theory by leveraging existing generalization bounds (e.g., see Corollary 2 in [HSKM22] for a link between 'lower' tails and generalization error). Secondly, it would be of great interest to extend our results to the non-convex deep learning settings. Finally, since this is a theoretical paper studying online and offline SGD, our work contains no direct potential negative societal impacts.
Acknowledgments. We thank Benjamin Dupuis for the valuable feedback. AD would like to thank the Isaac Newton Institute for Mathematical Sciences for support and hospitality during the programme The mathematical and statistical foundation of future data-driven engineering when work on this paper was undertaken. Umut Şimsekli's research is supported by the French government under management of Agence Nationale de la Recherche as part of the "Investissements d'venir" program, reference ANR-19-P3IA-0001 (PRAIRIE 3IA Institute) and the European Research Council Starting Grant DYNASTY - 101039676.
The monotonic relation between the tail exponent and the  $\eta /b$  ratio is in line with findings from [GSZ21], although new findings (see Sec.6.6 in [ZLMU22]) point out that a mere dependence on this ratio has been found broken. We believe that for other ranges of the parameters (outside of the ones originally used in [GSZ21]), a different behavior could be observed.
[RZGS23] Anant Raj, Lingjiong Zhu, Mert Gurbuzbalaban, and Umut Şimsekli. Algorithmic stability of heavy-tailed sgd with general loss functions. arXiv preprint arXiv:2301.11885, 2023.
[SCW20] James Stewart, Daniel K Clegg, and Saleem Watson. *Calculus: early transcendents*. Cengage Learning, 2020.
$[\S \mathrm{GN}^{+}19]$  Umut Şimsekli, Mert Gurbuzbalaban, Thanh Huy Nguyen, Gael Richard, and Levent Sagun. On the heavy-tailed theory of stochastic gradient descent for deep neural networks. arXiv preprint arXiv:1912.00018, 2019.
[SSDE20] Umut Simsekli, Ozan Sener, George Deligiannidis, and Murat A Erdogdu. Hausdorff dimension, heavy tails, and generalization in neural networks. Advances in Neural Information Processing Systems, 33:5138-5151, 2020.
[SSG19] Umut Simsekli, Levent Sagun, and Mert Gurbuzbalaban. A tail-index analysis of stochastic gradient noise in deep neural networks. In International Conference on Machine Learning, pages 5827-5837. PMLR, 2019.
[Vil09] Cédric Villani. The wasserstein distances. In Optimal transport, pages 93-111. Springer, 2009.
$\left[\mathrm{ZFM}^{+}20\right]$  Pan Zhou, Jiashi Feng, Chao Ma, Caiming Xiong, Steven Chu Hong Hoi, et al. Towards theoretically understanding why sgd generalizes better than adam in deep learning. Advances in Neural Information Processing Systems, 33:21285-21296, 2020.
[ZLMU22] Liu Ziyin, Kangqiao Liu, Takashi Mori, and Masahito Ueda. Strength of minibatch noise in sgd, 2022.
The organization of the appendix is as follows:
- In Section A, we provide the background material required for the proof methodology.
- In Section B, we present technical results used to conclude the results of our work.
- In Section C, we give the proofs of Theorems 3-6.
- In Section D, we provide additional experimental results.
- In Section E, we justify the tail estimator choice.
A Additional Technical Background
A.1 Existing results
Lemma 1 (Explicit solution of Wasserstein distance [PZ19]). Let  $\mu_1, \mu_2 \in \mathcal{P}(\mathbb{R})$  be two probability measures on  $\mathbb{R}$ , and denote their cumulative distribution functions by  $F_1(x)$  and  $F_2(x)$  respectively. Then, the Wasserstein- $p$  distance between  $\mu_1$  and  $\mu_2$  has an explicit formula:
$$
\mathbf {W} _ {p} \left(\mu_ {1}, \mu_ {2}\right) = \left(\int_ {0} ^ {1} \left| F _ {1} ^ {- 1} (q) - F _ {2} ^ {- 1} (q) \right| ^ {p} \mathrm {d} q\right) ^ {1 / p},
$$
where  $F_1^{-1}$  and  $F_2^{-1}$  denote the quantile functions. In the case when  $p = 1$ , by applying the change of variables, one can obtain the following:
$$
\mathbf {W} _ {1} \left(\mu_ {1}, \mu_ {2}\right) = \int_ {\mathbb {R}} \left| F _ {1} (x) - F _ {2} (x) \right| d x.
$$
Now, for  $q > 0, \alpha > 0, \gamma > 0$  and  $\mu \in \mathcal{P}\left(\mathbb{R}^d\right)$ , let:
$$
M _ {q} (\mu) := \int_ {\mathbb {R} ^ {d}} | x | ^ {q} \mu (\mathrm {d} x) \text {a n d} \mathcal {E} _ {\alpha , \gamma} (\mu) := \int_ {\mathbb {R} ^ {d}} \mathrm {e} ^ {\gamma | x | ^ {\alpha}} \mu (\mathrm {d} x).
$$
Furthermore, consider an i.i.d. sequence  $(X_{k})_{k\geq 1}$  of  $\mu$  -distributed random variables and, for  $n\geq 1$  define the empirical measure by:
$$
\mu^ {(n)} := \frac {1}{n} \sum_ {k = 1} ^ {n} \delta_ {X _ {k}}.
$$
We can now proceed to state the following result.
Lemma 2 ([FG15]). Let  $\mu \in \mathcal{P}\left(\mathbb{R}^d\right)$  and let  $p > 0$ . Assume one of the three following conditions:
(1)  $\alpha > p, \gamma > 0, \mathcal{E}_{\alpha, \gamma}(\mu) < \infty,$
(2)  $\alpha \in (0,p),\gamma >0,\mathcal{E}_{\alpha ,\gamma}(\mu) <   \infty$
(3)  $q > 2p, M_q(\mu) < \infty$ .
Then for all  $n\geq 1$  ,all  $x\in (0,\infty)$
$$
\mathbb {P} \left(\mathbf {W} _ {p} (\mu^ {(n)}, \mu) \geq x\right) \leq a (n, x) \mathbf {1} _ {\{x \leq 1 \}} + b (n, x)
$$
$$
a (n, x) = C \left\{ \begin{array}{l l} \exp \left(- c n x ^ {2}\right) & i f p > d / 2 \\ \exp \left(- c n (x / \log (2 + 1 / x)) ^ {2}\right) & i f p = d / 2 \\ \exp \left(- c n x ^ {d / p}\right) & i f p \in [ 1, d / 2) \end{array} \right.
$$
$$
b (n, x) = C \left\{ \begin{array}{l l} \exp \left(- c n x ^ {\alpha / p}\right) \mathbf {1} _ {\{x > 1 \}} & \text {u n d e r (1)}, \\ \exp \left(- c (n x) ^ {(\alpha - \varepsilon) / p}\right) \mathbf {1} _ {\{x \leq 1 \}} + \exp \left(- c (n x) ^ {\alpha / p}\right) \mathbf {1} _ {\{x > 1 \}} & \forall \varepsilon \in (0, \alpha) \text {u n d e r (2)}, \\ n (n x) ^ {- (q - \varepsilon) / p} & \forall \varepsilon \in (0, q) \text {u n d e r (3)}. \end{array} \right.
$$
The positive constants  $C$  and  $c$  depend only on  $p, d$  and either on  $\alpha, \gamma, \mathcal{E}_{\alpha, \gamma}(\mu)$  (under (1)) or on  $\alpha, \gamma, \mathcal{E}_{\alpha, \gamma}(\mu), \varepsilon$  (under (2)) or on  $q, M_q(\mu), \varepsilon$  (under (3)).
B Technical Lemmas
Lemma 3. Let  $X$  and  $Y$  be two random vectors in  $\mathbb{R}^d$ . Then, we have
$$
\mathbf {W} _ {1} (\mathcal {L} (X), \mathcal {L} (Y)) \geq \mathbf {W} _ {1} (\mathcal {L} (\| X \|), \mathcal {L} (\| Y \|)). \tag {17}
$$
Proof. By the definition of the Wasserstein distance, we have that
$$
\begin{array}{l} \mathbf {W} _ {1} (\mathcal {L} (X), \mathcal {L} (Y)) = \inf  _ {\gamma \in \Gamma (\mathcal {L} (X), \mathcal {L} (Y))} \int_ {\mathbb {R} ^ {d} \times \mathbb {R} ^ {d}} \| x - y \| \mathrm {d} \gamma (x, y) (18) \\ = \int_ {\mathbb {R} ^ {d} \times \mathbb {R} ^ {d}} \| x - y \| \mathrm {d} \gamma^ {*} (x, y), (19) \\ \end{array}
$$
where  $\gamma^{*}$  is the coupling that attains the infimum (the proof of the existence of such coupling for any Wasserstein- $p$  distance, where  $p \geq 1$ , can be found in, e.g., [Bog06, Theorem 8.10.45]). Then, by the reverse triangle inequality, we have
$$
\begin{array}{l} \mathbf {W} _ {1} (\mathcal {L} (X), \mathcal {L} (Y)) \geq \int_ {\mathbb {R} ^ {d} \times \mathbb {R} ^ {d}} | \| x \| - \| y \| | d \gamma^ {*} (x, y) (20) \\ = \int_ {\mathbb {R} _ {+} \times \mathbb {R} _ {+}} | x - y | \mathrm {d} T _ {\#} \gamma^ {*} (x, y), (21) \\ \end{array}
$$
where  $T: \mathbb{R}^d \times \mathbb{R}^d \mapsto \mathbb{R}_+ \times \mathbb{R}_+$  is the map  $(x,y) \mapsto (\| x\|, \| y\|)$  and  $T_{\#} \gamma^{*}$  is the pushforward measure such that for a measurable set  $B \subset \mathbb{R}_{+} \times \mathbb{R}_{+}$ , we have  $T_{\#} \gamma^{*}(B) = \gamma^{*}(T^{-1}(B))$ . As  $T_{\#} \gamma^{*}$  is a coupling between  $\| X\|$  and  $\| Y\|$ , we have that