SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In this section, we compare our theoretical predictions with the results of numerical experiments for a large family of data distributions. The results have been obtained using ridge  $\ell_2$ -regularisation, and both quadratic and logistic losses, with various data cluster balances  $\rho$ . We will also assume, without loss of generality, that  $\mu_{\pm} = \pm \frac{1}{\sqrt{d}}\mu$ , where  $\mu \sim \mathcal{N}(\mathbf{0},\mathbf{I}_d)$ . The synthetic data sets will be produced using, an inverse-Gamma-distributed variance  $\Delta$ , with density parametrised as
$$
\varrho (\Delta) \equiv \varrho_ {a, c} (\Delta) = \frac {c ^ {a}}{\Gamma (a) \Delta^ {a + 1}} e ^ {- \frac {c}{\Delta}}, \tag {11a}
$$
depending on the shape parameter  $a > 0$  and on the scale parameter  $c > 0$ . As a result, each class will be distributed as
$$
P (\boldsymbol {x} \mid \boldsymbol {\mu}) = \frac {(2 c) ^ {a} \Gamma (a + d / 2)}{\Gamma (a) \pi^ {d / 2} (2 c + \| \boldsymbol {x} - \boldsymbol {\mu} \| ^ {2}) ^ {a + d / 2}}, \tag {11b}
$$
which produces a cloud centered in  $\pmb{\mu}$ . The reason for our choice is that the distribution  $\varrho_{a,c}$  allows us to easily explore different regimes by changing two parameters only. Indeed, the distribution has, for  $a > 1$ , covariance matrix  $\Sigma = \mathbb{E}[(x - \mu)\otimes (x - \mu)^{\top}] = \sigma^{2}I_{d}$ , where  $\sigma^2 = \frac{c}{a - 1}$ . By fixing  $\sigma^2$  and taking the limit  $a\to +\infty$ ,  $P(x|\mu)\to \mathcal{N}(x|\mu ,\sigma^2 I_d)$ , i.e., the Gaussian case discussed by Mignacco et al. [49]. For  $a\in (0,1]$ , instead,  $\sigma^2 = +\infty$ . Note that the choice of inverse-Gamma distribution is not that unusual. It has been adopted, for example, to describe non-Gaussian data in quantitative finance [15, 40] or econometrics models [52]. Finally, we will construct our label estimator as in Eq. (2) with  $\varphi (x) = \mathrm{sign}(x)$ .
3.1 Finite-covariance case
Figure 1: Test error  $\epsilon_{g}$  (solid line, top), training error  $\epsilon_{t}$  (center) and training loss  $\epsilon_{\ell}$  (bottom) as predicted by Eq. (8) in the balanced  $\rho = 1 / 2$  case. The dataset distribution is parametrised as in Eq. (12). The classification task is solved using a quadratic loss with ridge regularisation with  $\lambda = 10^{-5}$ . In the top figure, the dashed line corresponds to the Bayes optimal bound. Dots correspond to the average outcome of 50 numerical experiments in dimension  $d = 10^{3}$ . In our parametrisation, the population covariance is  $\Sigma = I_d$  for all values of  $a$  and moreover, for  $a\to +\infty$ , the case of Gaussian clouds with the same centroids and covariance is recovered. For further details on the numerical solutions, see Appendix B.
Let us start by considering a data distribution as in Eq. (11) with shape parameter  $a = c + 1 > 1$ , which is therefore given by
$$
P (\boldsymbol {x} \mid \boldsymbol {\mu}) = \frac {2 ^ {a} (a - 1) ^ {a} \Gamma (a + d / 2)}{\Gamma (a) \pi^ {d / 2} (2 a - 2 + \| x - \boldsymbol {\mu} \| ^ {2}) ^ {a + d / 2}} \tag {12}
$$
and decays as  $\| x\|^{-2a - 1}$  in the radial direction for  $\| x - \pmb {\mu}\| \gg 0$  . As a consequence, the distribution has  $\lim_{d}{}^{1} / d\mathbb{E}[\| x - \pmb {\mu}\|^{k}] = +\infty$  for  $k\geq 2a$  . With this choice, all elements of the family in Eq. (12) have the same covariance  $\pmb {\Sigma} = \mathbb{E}[(\pmb {x} - \pmb {\mu})\otimes (\pmb {x} - \pmb {\mu})^{\top}] = \pmb{I}_{d}$  as  $a$  is varied, including the Gaussian limit  $P(x|\pmb {\mu})\to$ $\mathcal{N}(\pmb {\mu},\pmb{I}_d)$  obtained for  $a\rightarrow +\infty$
In Fig. 1 we present the results of our numerical experiments using the square loss and small regularisation. An excellent agreement between the theoretical predictions and the results of numerical experiments is found for a range of values of  $a$  and sample complexity  $\alpha$ , both for balanced, i.e., equally sized, and unbalanced clusters of data (the plot for this case can be found in Appendix A.3). The test error  $\epsilon_{g}$  presents the classical interpolation peak at  $\alpha = 1$ , smoothened by the presence of a non-zero regularisation strength  $\lambda$ , and the typical double-descent behavior [54]. As  $a$  is increased, the results of Mignacco et al. [49] for Gaussian clouds with  $P(\boldsymbol{x}|\boldsymbol{\mu}) = \mathcal{N}(\boldsymbol{x}|\boldsymbol{\mu},\boldsymbol{I}_d)$  are approached. The  $a\rightarrow +\infty$  curves correspond, therefore, to the Gaussian mixture model we would have constructed by simply fitting the first and second moments of each class in the finite- $a$  case. The plot shows that, at given population covariance  $\Sigma = I_d$ , the classification of power-law distributed clouds is associated with different, and in particular smaller, errors than the corresponding task on Gaussian clouds with the same covariance: in this sense, no Gaussian
universality principle holds. This (perhaps) counter-intuitive effect stems from the fact that, although for all classes  $\pmb{\Sigma} = \pmb{I}_d$ , in our example the mean absolute deviation is smaller for finite  $a$
$$
\lim  _ {d \rightarrow + \infty} \frac {\mathbb {E} [ \| x - \mu \| ]}{\sqrt {\mathbb {E} [ \| x - \mu \| ^ {2} ]}} = \frac {\Gamma (a - 1 / 2) \sqrt {a - 1}}{\Gamma (a)} \xrightarrow {a \rightarrow + \infty} 1 ^ {-}. \tag {13}
$$
The learning process benefits from the fact that as  $a$  decreases while keeping the variance fixed, points move on average closer to their means  $\mu_{\pm}$  to compensate heavier tails. Finally, we observed that test errors are systematically smaller for unbalanced clusters than for balanced clusters (see Appendix A.3).
The same numerical experiment was repeated using the logistic loss for training. Once again, in Fig. 2, we focus on  $\rho = 1/2$  and show that the theoretical predictions of the generalisation and training errors agree with the results of numerical experiments for a range of sample complexity values  $\alpha$  and various values of  $a$ . Just as for the square loss, the test error  $\epsilon_{g}$  of the Gaussian model  $(a \to +\infty)$  is larger than the one observed for power-law distributed clouds at finite values of  $a$ . In the  $\lambda \to 0$  limit, the typical interpolation cusp in the generalisation error is observed: the cusp occurs at different values of  $\alpha$  as  $a$  is changed; interpolation occurs at smaller values of  $\alpha$  for larger  $a$  (i.e., for "more Gaussian" distributions). A comparison with the test error  $\epsilon_{t}$  confirms that this cusp occurs at the value of  $\alpha$  where the training error becomes non-zero and the two training data clouds become non-separable [70, 16]. This sharp transition in  $\alpha$  also corresponds to the existence transition of maximum-likelihood estimator in high-dimensional logistic regression, analysed for Gaussian data in Refs. [70, 49] and, in our setting, in Section 4 below. For larger regularisation strength, the cusp smoothens, and the training error becomes non-zero at smaller values of  $\alpha$ .
3.2 Infinite-covariance case
The family of distributions specified by Eq. (11) allows us to also consider the case of infinite covariance, i.e.,  $\sigma^2 = +\infty$ . To test our formulas in this setting, we considered the case in which each cloud is obtained by "contaminating" a standard Gaussian with an infinite-covariance distribution as in Eq. (11b) with  $c = 1$  and  $a < 1$  [35]. In other words, we use the density  $\varrho(\Delta) = r \varrho_{a,1}(\Delta) + (1 - r) \delta(\Delta - 1)$ , with  $r \in [0,1]$  interpolation parameter, for  $\Delta$  in Eq. (1). Each class has therefore infinite covariance for  $0 < r \leq 1$  and the Gaussian case is recovered for  $r = 0$ . Fig. 3 collects our results for two balanced clouds, analysed using square loss and logistic loss. Good agreement between the theoretical predictions and the results of numerical experiments is found for a range of values of sample complexity and of the ratio  $r$ . In this case, the finite-variance case  $r = 0$  corresponds, not surprisingly, to the lowest test error with both square loss and logistic loss, which grows with  $r$ .
3.3 The role of regularisation in the classification of non-Gaussian clouds
Figure 2: Test error  $\epsilon_{g}$  (top), training error  $\epsilon_{t}$  (center) and training loss  $\epsilon_{\ell}$  (bottom) via logistic loss training on balanced clusters parametrised as in Eq. (12) ( $\Sigma = I_d$ ). Ridge regularisation with  $\lambda = 10^{-4}$  is adopted. Dots correspond to the average over 20 numerical experiments with  $d = 10^3$ . The Gaussian limit is recovered for  $a \to +\infty$ . Further details on the numerical solutions can be found in Appendix B.
One of the main results in the work of Mignacco et al. [49] is related to the effect of regularisation on the performances in a classification task on a Gaussian mixture. They observed that, for all values of the sample complexity  $\alpha$ , the optimal ridge classification performance on a pair of balanced clouds is obtained for an infinite regularisation strength  $\lambda$ , using both hinge loss and square loss. We tested the robustness of this result beyond the purely Gaussian setting. In Fig. 4 (left), we plot the test error obtained using square loss and different regularisation strengths  $\lambda$  on the dataset distribution as in Eq. (12) with  $a = 2$ . We observe that, both in the balanced case and in the unbalanced case, the optimal regularisation strength  $\lambda^{\star}$  is finite and, moreover,  $\alpha$ -dependent. On the other hand, if we fix  $\alpha$  and let  $a$  grow towards  $+\infty$ , the optimal  $\lambda^{\star}$  grows as well to recover the result of Mignacco et al. [49] of diverging optimal regularisation for balanced Gaussian clouds, as shown in Fig. 4 (center)
Figure 3: Test error  $\epsilon_{g}$  in the classification of two balanced clouds, via quadratic loss (left) and logistic loss (right). In both cases, ridge regularisation is adopted ( $\lambda = 10^{-4}$  for the square loss case,  $\lambda = 10^{-3}$  for the logistic loss case). Each cloud is a superposition of a power-law distribution with infinite variance and a Gaussian with covariance  $\pmb{\Sigma} = \pmb{I}_d$ . The parameter  $r$  allows us to contaminate the purely Gaussian case ( $r = 0$ ) with an infinite-variance contribution ( $0 < r \leq 1$ ) as in Eq. (11) with  $c = 1$  and  $a = 1/2$  (left) or  $a = 3/4$  (right). Dots correspond to the average test error of 20 numerical experiments in dimension  $d = 10^3$ . Note that, at a given sample complexity, Gaussian clouds are associated with the lowest test error for both losses.
for  $\alpha = 2$ . This result is represented also in Fig. 4 (right), where it is shown that, in the unbalanced case, the optimal regularisation strength  $\lambda^{\star}$  instead saturates to a finite value for  $a \to +\infty$ , i.e., for Gaussian clouds [49].
Figure 4: (Left) Test error for ridge regularised quadratic loss for various regularisation strengths. The data points of each cloud in the training set are distributed as in Eq. (12), with shape parameter  $a = 2$ , for balanced clusters (top) and unbalanced clusters ( $\rho = 1/4$ , bottom). Points are the results of 50 numerical experiments, and the dashed lines are Bayes-optimal bounds. (Center) Test error for different regularisation strengths  $\lambda$  for two balanced clusters with quadratic loss at sample complexity  $\alpha = 2$  using the data distribution (12). The optimal regularisation strength value  $\lambda^{\star}$  obtained from averaging 5 runs for each  $a$  is marked with a cross. (Right) Optimal regularisation strength  $\lambda^{\star}$  at  $\alpha = 2$  for different values of  $a \in [1.5, 10^{2}]$  for both balanced and unbalanced clusters, obtained from averaging 5 runs. Note that, for  $\rho = 1/2$ ,  $\lambda^{\star} \to +\infty$  as  $a \to +\infty$ .
4 The separability threshold
By studying the training error  $\epsilon_t$  with logistic loss at zero regularisation we can obtain information on the boundary between the regime in which the training data are perfectly separable and the regime in which they are not. In other words, we can extract the value of the so-called separability transition complexity  $\alpha^{\star}$  [70, 49, 43]. Once again, a complete characterisation of this transition point is available in the Gaussian mixture case, where  $\alpha^{\star}$  can be explicitly given as a function of  $\sigma^2$  [49].
It is not trivial to extend this result to the general case of non-Gaussian mixtures. It is, however, possible to derive  $\alpha^{\star}$  within the large family of "superstatistical" models we are considering here under the assumption that Result 2.1 holds. In this case, in Appendix A.3 we prove the following.
Result 4.1 In the considered double-stochastic model, data are linearly separable for  $\alpha < \alpha^{\star}$ , where
$$
\alpha^ {\star} := \max  _ {\theta \in (0, 1 ], \gamma} \frac {1 - \theta^ {2}}{\mathcal {S} (\theta , \gamma)}, \quad \mathcal {S} (\theta , \gamma) := \int_ {0} ^ {\infty} f ^ {2} \mathbb {E} _ {\Delta} \left[ \rho_ {+} \mathcal {N} \left(f + \frac {\theta + \gamma}{\sqrt {\Delta}} | 0, 1\right) + \rho_ {-} \mathcal {N} \left(f + \frac {\theta - \gamma}{\sqrt {\Delta}} | 0, 1\right) \right] d f. \tag {14}
$$
Fig. 5 shows the values of  $\alpha^{\star}$  for different choices of the shape parameters  $a$  and  $c$  in the case of two balanced clouds of datapoints distributed as in Eq. (11b) as predicted by Eq. (14). In Fig. 5 (left), we fixed  $\sigma^2 = \frac{c}{a - 1} < +\infty$ , and plotted the separability threshold  $\alpha^{\star}$  for a range of values of the shape parameter  $a > 1$ . As  $a$  grows, the known threshold value for the Gaussian mixture case is recovered [49]. The double limit  $a \to \infty$  and  $\sigma^2 \to \infty$ , on the other hand, provides the Cover [12] transition value  $\alpha^{\star} = 2$ , as expected. In Fig. 5 (right), instead, we analyse the case of infinite-variance clusters, by fixing  $a \in (0,1)$  and by varying the scale parameter  $c$  in Eq. (11b), which controls the spread (width) of the distribution. The Cover transition is therefore correctly recovered as  $c$  diverges for all values of  $a$ .
Figure 5: Separability threshold  $\alpha^{\star}$  obtained by solving the equations in Eq. (7) with logistic loss, ridge regularisation strength  $\lambda = 10^{-5}$  and  $\rho = 1/2$ . The data points of each cloud are distributed around their mean  $\mu$  with a two-parameter distribution as in Eq. (11b). (Left). Finite covariance  $\Sigma = \sigma^2 I_d$  case,  $\sigma^2 = \frac{c}{a - 1}$ , as a function of  $a$ . Dashed lines are the threshold values of the Gaussian case derived by Mignacco et al. [49]. At large  $a$  and large variance  $\sigma^2$ , the Cover's transition  $\alpha = 2$  for balanced clusters is recovered. (Right) Infinite-variance data clouds obtained by fixing  $0 < a < 1$  in Eq. (11b).
5 Random labels in the teacher-student scenario and Gaussian universality
Our analysis can be extended to the case in which a point  $x$  is labeled not according to its class but by a "teacher", represented by a probabilistic law  $P_0(y|\tau)$  and a vector  $\theta_0 \in \mathbb{R}^d$  so that each dataset element is independently generated with the joint distribution
$$
P (\boldsymbol {x}, y) = P _ {0} \left(y \left| \frac {\theta_ {0} ^ {\intercal} \boldsymbol {x}}{\sqrt {d}} \right.\right) \sum_ {k = \pm} \rho_ {k} \mathbb {E} \left[ \mathcal {N} \left(\boldsymbol {x} \mid \boldsymbol {\mu} _ {k}, \Delta \boldsymbol {I} _ {d}\right) \right]. \tag {15}
$$
Let us assume now that  $\pmb{\theta}_0$  is such that  $\lim_{d}1 / \sqrt{a}\pmb{\theta}_{0}^{\top}\pmb{\mu}_{\pm} = 0$ , i.e., the teacher is "uncorrelated" with the structure of the mixture. In Appendix A.5 we show that, if  $\ell (y,\eta) = \ell (-y, - \eta)$  and  $P_0(y|\tau) = P_0(-y| - \tau)$ , the task is equivalent to a classification problem on a single cloud centered in the origin, i.e., to the case  $\pmb{\mu}_{\pm} = \mathbf{0}$ , so that  $P(x,y) = P_0(y|^1 /\sqrt{a}\pmb{\theta}_0^\top x)\mathbb{E}[\mathcal{N}(\pmb {x}|\pmb {0},\Delta I_d)]$ . This mean-invariance, recently proven in Ref. [25] in the pure Gaussian setting, holds therefore in our non-Gaussian setting as well.
A special, relevant case of "uncorrelated teacher" is the one of random labels, where  $P_0(y|\tau) = \frac{1}{2} (\delta_{y, + 1} + \delta_{y, - 1})$ . This apparently vacuous model is relevant in the study of the storage capacity problem [21, 39, 11] and of the separability threshold [12], and it has also been an effective tool to better understand deep learning [45, 75]. In this case, the following universal result holds with respect to our setup.
Result 5.1 In the random label setting studied via ridge regression case with zero regularisation, the asymptotic training loss is independent of the distribution of  $\Delta$ , and given by the universal formula
$$
\frac {1}{2 n} \sum_ {\nu = 1} ^ {\infty} \left(y ^ {\nu} - \frac {\boldsymbol {x} ^ {\top} \boldsymbol {w} ^ {\star}}{\sqrt {d}}\right) ^ {2} \xrightarrow [ n / d = \alpha ]{n \rightarrow + \infty} \frac {\alpha - 1}{2 \alpha} \theta (\alpha - 1) =: \epsilon_ {\ell}. \tag {16}
$$
The proof of the previous result is given in Appendix A.5. Note that this result does not require  $\sigma^2 \coloneqq \mathbb{E}[\Delta]$  to be finite, and it includes, as a special case, the Gaussian mixture model with random labels considered in Ref. [25, 59]. Therein, Eq. (16) has been obtained under the Gaussian design hypothesis. It turns out therefore that Eq. (16) holds for a much larger ensemble, which includes heavy-tailed distributions with possibly infinite covariance. On the other hand, universality does not hold, e.g., for the training error.
In Fig. 6 we exemplify these results by using ridge-regularised square loss. The numerical experiments correspond to a classification task on a dataset of two clouds centered in  $\pmb{\mu}_{\pm}$ ,  $\pmb{\mu}_{+} = -\pmb{\mu}_{-} \sim \mathcal{N}(\mathbf{0}, \frac{1}{d}\pmb{I}_d)$ , obtained using the distribution
$$
P (\boldsymbol {x}, y) = \frac {\delta_ {y , + 1} + \delta_ {y , - 1}}{4} \sum_ {k = \pm} P (\boldsymbol {x} \mid \boldsymbol {\mu} _ {k}), \tag {17}
$$
where  $P(\pmb{x}|\pmb{\mu}_{\pm})$  is the distribution in Eq. (11b) depending on the shape parameters  $a$  and  $c$ . These results are compared with the theoretical prediction obtained for a single cloud, with the same parameters  $a$  and  $c$ , but centered in the origin, so that  $P(\pmb{x},y) = \frac{\delta_{y, + 1} + \delta_{y, - 1}}{2} P(\pmb{x}|\pmb{\mu} = \mathbf{0})$ : the agreement confirms the "mean-invariance" property stated above. As anticipated, as we are considering random labels, the training loss  $\epsilon_{\ell}$  follows Eq. (16) in all cases, whereas the test and training errors are not universal. Note that, in place of the test error (which is trivially  $\epsilon_g = 1 / 2$ ) we plot the mean square error on the pre-activations  $\hat{\epsilon}_g \coloneqq \mathbb{E}_{(y,x)}[(y - 1/\sqrt{d}\pmb{x}^\top\pmb{w}^\star)^2]$ . Also, observe that this quantity is divergent in the case of infinite covariance, hence is absent from the plot for  $a \in (0,1]$ .
6 Conclusions and perspectives
The framework introduced in this work has allowed us to perform an exact asymptotic analysis of the learning of a mixture of two data point clouds with possibly power-law tails. In our dataset, each sample point  $\pmb{x}$  is labeled by  $y \in \{-1, 1\}$  and obtained as  $\pmb{x} = y\pmb{\mu} + \sqrt{\Delta} z$ , where  $\pmb{\mu} \in \mathbb{R}^d$ ,  $z \sim \mathcal{N}(\pmb{0}, \pmb{I}_d)$  and  $\Delta \sim \varrho$ . We have derived exact expressions for the test and training errors and training loss achieved by a GLM in the limit of large size  $n$  of the dataset and large dimensionality  $d$  under the assumption that  $\alpha = {}^n / d$  is kept fixed. We have shown that the performance crucially depends on higher-order moments of the data distribution, so that a Gaussian universality assumption is not feasible. Moreover, we have shown the effects of non-Gaussianity on the optimal regularisation strength and derived an exact formula for the critical  $\alpha^\star$  at which the two clouds become linearly inseparable. Finally, we have considered the so-called random labels setting (in which sample label assignment is randomised) and shown that, in this case, the training square loss takes a universal asymptotic form independent of the distribution of  $\Delta$ : this result further supports the crucial role of labels' randomness in some recent universality results [25].
In conclusion, we have provided an exactly solvable toy model exhibiting power-law tails and possibly unbounded covariance, highlighting the fact that a dataset deviation from Gaussianity in high dimensions cannot, in general, be neglected. The application of the introduced theoretical framework to real datasets is an interesting direction to explore in the future: the main difficulty, in this case, is the choice of the best distribution  $\varrho$  given the observed dataset, a problem that has a long tradition in the context of Bayesian estimation [2, 23].
Figure 6: Mean square error  $\hat{\epsilon}_{g}$  (top), training error  $\epsilon_{t}$  (center) and training loss  $\epsilon_{\ell}$  (bottom) obtained using square loss on a dataset distributed as in Eq (17) with random labels (dots) compared with the theoretical predictions for the single cluster case (continuous lines). The distributions are parametrised using Eq. (12) for  $a > 1$  and Eq. (11b) for  $0 < a \leq 1$ . A ridge regularisation with  $\lambda = 10^{-4}$  is adopted. Note that  $\hat{\epsilon}_{g}$  is absent in the top plot for  $a = 1/2$  and  $c = 1$  being a divergent quantity in this case.
The authors would like to thank Nikolas Nusken for stimulating discussions. Also, we would like to thank Bruno Loureiro for his feedback and for his careful reading of the manuscript.
In a more simplistic approach, it can be observed that in the case in which the square loss is adopted, equations (10) depend on  $\mathbb{E}[(1 + \nu \Delta)^{-1}]$  and  $\mathbb{E}[(1 + \nu \Delta)^{-2}]$  only. Such quantities can be numerically estimated from the dataset, for example, by empirically evaluating the identity  $\mathbb{E}[(1 + \nu \Delta)^{-1}] = \sqrt{\nu}\partial_{\nu}\int_{0}^{\nu}(\nu -u)^{-1 / 2}\mathbb{E}[\exp (-ux^{2} / 2)]\mathrm{d}u$  when it is assumed that  $x\sim \mathbb{E}[\mathcal{N}(0,\Delta)]$