SlowGuess/NeurIPS_2023_OCR huggingface dataset:
(c)  $(n,p) = (5000,1500)$
Figure 1: Scatter plot of pairs  $(\sqrt{n}\hat{A}_{j1},\sqrt{n}\hat{A}_{j2})$  with  $K = 2$  over 1000 repetitions. The blue ellipsoid is the boundary of the  $95\%$  confidence set for  $\sqrt{n}\hat{A}^T e_j$  under  $H_0$  from the classical MLE theory (1.12)-(1.13) based on the Fisher information, the dashed red ellipsoids are the boundaries of the  $95\%$  confidence set for  $\sqrt{n}\hat{A}^T e_j$  under  $H_0$  from this paper (cf. (2.3) below). Each of the 1000 repetition gives a slightly different dashed ellipsoid. The solid red ellipsoid is the average of these 1000 dashed ellipsoids. Each row of  $X$  is i.i.d. sampled from  $N(\mathbf{0},\Sigma)$  with  $\Sigma = (0.5^{|i - j|})_{p\times p}$ . The first  $\lceil p / 4\rceil$  rows of  $A^{*}$  are i.i.d. sampled from  $N(\mathbf{0},I_K)$  while other rows are set to zeros. We further normalize  $A^{*}$  such that  $A^{*T}\Sigma A^{*} = I_{K}$ . The last coordinate  $j = p$  is used as the null coordinate.
- Establish asymptotic normality of the multinomial MLE  $\hat{A}^T e_j$  for null covariates as  $n, p \to +\infty$  simultaneously with a finite limit for  $n/p$ .
- Develop a valid methodology for hypothesis testing of (1.10) in this regime, i.e., testing for the presence of an effect of a feature  $j \in [p]$  on the multiclass response.
The contribution of this paper is two-fold: (i) For a null covariate  $j \in [p]$ , we establish asymptotic normality results for  $\hat{A}^T e_j$  that are valid in the high-dimensional regime where  $n$  and  $p$  have the same order; (ii) we propose a user-friendly test for assessing the significance of a feature in multiclass classification problems.
2 Main result: asymptotic normality of  $\hat{B}^T e_j$  and  $\hat{A}^T e_j$  on null covariates
In this section, we present the main theoretical results of our work and discuss their significance. We work under the following assumptions.
Assumption 2.1. For constants  $\delta > 1$ , assume that  $n, p \to \infty$  with  $p/n \leq \delta^{-1}$ , and that the design matrix  $X \in \mathbb{R}^{n \times p}$  has  $n$  i.i.d. rows  $(x_i)_{i \in [n]} \sim N(\mathbf{0}, \Sigma)$  for some invertible  $\Sigma \in \mathbb{R}^{p \times p}$ . The observations  $(x_i, y_i)_{i \in [n]}$  are i.i.d. and each  $y_i$  is of the form  $y_i = f(U_i, x_i^T \mathsf{B}^*)$  for some deterministic function  $f$ , deterministic matrix  $\mathsf{B}^* \in \mathbb{R}^{p \times (K + 1)}$  such that  $\mathsf{B}^* \mathbf{1}_{K + 1} = \mathbf{0}_p$ , and latent random variable  $U_i$  independent of  $x_i$ .
Assumption 2.2 (One-hot encoding). The response matrix  $Y$  is in  $\mathbb{R}^{n \times (K + 1)}$ . Its  $i$ -th row  $y_i$  is a one-hot encoded vector, that is, valued in  $\{0,1\}^{K + 1}$  with  $\sum_{k = 1}^{K + 1}y_{ik} = 1$  for each  $i \in [n]$ .
The model  $y_{i} = f(U_{i},x_{i}^{T}\mathsf{B}^{*})$  for some deterministic  $f$  and  $\mathsf{B}^*$  and latent random variable  $U_{i}$  in Assumption 2.1 is more general than a specific generative model such as the multinomial logistic conditional probabilities in (1.4), as broad choices for  $f$  are allowed. In words, the model  $y_{i} = f(U_{i},x_{i}^{T}\mathsf{B}^{*})$  with  $\mathsf{B}^*\mathbf{1}_{K + 1} = 0_p$  means that  $y_{i}$  only depends on  $x_{i}$  through a  $K$  dimensional projection of  $x_{i}$  (the projection on the row-space of  $\mathsf{B}^*$ ). The assumption  $p / n\leq \delta^{-1}$  is more general than assuming a fixed limit for the ratio  $p / n$ ; this allows us to cover low-dimensional settings satisfying  $p / n\to 0$  as well.
The following assumption requires the labels to be "balanced": we observe each class at least  $\gamma n$  times for some constant  $\gamma > 0$ . If  $(y_{i})_{i \in [n]}$  are i.i.d. as in Assumption 2.1 with distribution independent of  $n, p$ , by the law of large numbers this assumption is equivalent to  $\min_{k \in [K + 1]} \mathbb{P}(y_{ik} = 1) > 0$ .
Assumption 2.3. There exists a constant  $\gamma \in (0, \frac{1}{K + 1}]$ , such that for each  $k \in [K + 1]$ , with probability approaching one at least  $\gamma n$  observations  $i \in [n]$  are such that  $y_{ik} = 1$ . In other words,  $\mathbb{P}\left(\sum_{i=1}^{n} I(y_{ik} = 1) \geq \gamma n\right) \to 1$  for each  $k \in [K + 1]$ .
As discussed in item list (i) on page 2, in binary logistic regression, Candès and Sur [2020], Sur and Candès [2019] show that the plane  $\left(\frac{p}{n},\| \Sigma^{1 / 2}\beta^{*}\|\right)$  is split by a smooth curve into two connected open components: in one component the MLE does not exist with high probability, while in the other component, with high probability the MLE exists and is bounded in the sense that  $\| \Sigma^{1 / 2}\hat{\beta}\| ^2 < \tau '$  or equivalently  $\frac{1}{n}\| X\hat{\beta}\| ^2 < \tau$  for constants  $\tau ,\tau^{\prime}$  independent of  $n,p$ . The next assumption requires the typical situation of the latter component, in the current multiclass setting:  $\hat{\mathbf{B}}$  in (1.5) exists in the sense that the minimization problem has solutions, and at least one solution is bounded.
Assumption 2.4. Assume  $\mathbb{P}(\hat{\mathsf{B}}$  exists and  $\| X\hat{\mathsf{B}} (I_{K + 1} - \frac{\mathbf{11}^T}{K + 1})\| _F^2\leq n\tau)\to 1$  as  $n,p\to +\infty$  for some large enough constant  $\tau$ .
Note that the validity of Assumption 2.4 can be assessed using the data at hand; if a multinomial regression solver (e.g. sklearn.linear_model.LogisticRegression) converges<sup>1</sup> and  $\frac{1}{n}\| X\hat{B}(I_{K+1} - \frac{11^T}{K+1})\|_F^2$  is no larger than a predetermined large constant  $\tau$ , then we know Assumption 2.4 holds. Otherwise the algorithm does not converge or produces an unbounded estimate: we know Assumption 2.4 fails to hold and we need collect more data.
<sup>1</sup>Here, we refer to standard convergence assessment methods for convex solvers, e.g., looking at the gradient/Hessian values at the current iterate, or looking at the duality gap if available.
Our first main result, Theorem 2.1, provides the asymptotic distribution of  $\hat{\mathsf{B}}^T e_j$  where  $j\in [p]$  is a null covariate, where  $\hat{\mathsf{B}}$  is any minimizer  $\hat{\mathsf{B}}$  of (1.5). Throughout, we denote by  $\Omega$  the precision matrix defined as  $\Omega = \Sigma^{-1}$ .
Theorem 2.1. Let Assumptions 2.1 to 2.4 be fulfilled. Then for any  $j \in [p]$  such that  $H_0$  in (1.10) holds, and any minimizer  $\hat{\mathsf{B}}$  of (1.5), we have
$$
\underbrace {\sqrt {\frac {n}{\Omega_ {j j}}}} _ {\text {s c a l a r}} \left(\underbrace {\left(\frac {1}{n} \sum_ {i = 1} ^ {n} \mathbf {g} _ {i} \mathbf {g} _ {i} ^ {T}\right) ^ {1 / 2}} _ {\mathbb {R} ^ {(K + 1) \times (K + 1)}}\right) ^ {\dagger} \underbrace {\left(\frac {1}{n} \sum_ {i = 1} ^ {n} \mathbf {V} _ {i}\right)} _ {\mathbb {R} ^ {(K + 1) \times (K + 1)}} \underbrace {\hat {\mathsf {B}} ^ {T} e _ {j}} _ {\mathbb {R} ^ {K + 1}} \xrightarrow {\mathrm {d}} N \left(\mathbf {0}, \underbrace {I _ {K + 1} - \frac {\mathbf {1 1} ^ {T}}{K + 1}} _ {\text {c o v .} \mathbb {R} ^ {(K + 1) \times (K + 1)}}\right), \tag {2.1}
$$
where  $\mathsf{g}_i = -\mathsf{y}_i + \hat{\mathsf{p}}_i$  as in (1.7) and  $\mathsf{V}_i = \mathsf{H}_i - (\mathsf{H}_i \otimes x_i^T)[\sum_{l=1}^{n} \mathsf{H}_l \otimes (x_l x_l^T)]^\dagger (\mathsf{H}_i \otimes x_i)$ .
The proof of Theorem 2.1 is given in Supplementary Section S3. Theorem 2.1 establishes that under  $H_0$ ,  $\hat{\mathbf{B}}^T e_j$  converges to a singular multivariate Gaussian distribution in  $\mathbb{R}^{K+1}$ . In (2.1), the two matrices  $\frac{1}{n} \sum_{i=1}^{n} (\mathsf{y}_i - \hat{\mathsf{p}}_i)(\mathsf{y}_i - \hat{\mathsf{p}}_i)^T$  and  $\frac{1}{n} \sum_{i=1}^{n} \mathsf{V}_i$  are symmetric with kernel being the linear span of  $\mathbf{1}_{K+1}$ , and similarly, if a solution exists, we may replace  $\hat{\mathbf{B}}$  by  $\hat{\mathbf{B}}(I_{K+1} - \frac{\mathbf{11}^T}{K+1})$  which is also solution in (1.5). In this case, all matrix-matrix and matrix-vector multiplications, matrix square root and pseudo-inverse in (2.1) happen with row-space and column space contained in the orthogonal component of  $\mathbf{1}_{K+1}$ , so that the limiting Gaussian distribution in  $\mathbb{R}^{K+1}$  is also supported on this  $K$ -dimensional subspace.
Since the distribution of the left-hand side of (2.1) is asymptotically pivotal for all null covariates  $j \in [p]$ , Theorem 2.1 opens the door of statistical inference for multinomial logistic regression in high-dimensional settings. By construction, the multinomial logistic estimate  $\hat{A} \in \mathbb{R}^{p \times K}$  in (1.9) ensures  $(\hat{A}, \mathbf{0}_p)$  is a minimizer of (1.5). Therefore, we can deduce the following theorem from Theorem 2.1.
Theorem 2.2. Define the matrix  $R = (I_K, \mathbf{0}_K)^T \in \mathbb{R}^{(K + 1) \times K}$  using block matrix notation. Let Assumptions 2.1 to 2.4 be fulfilled. For  $\hat{A}$  in (1.9) and any  $j \in [p]$  such that  $H_0$  in (1.10) holds,
$$
\underbrace {\left(I _ {K} + \frac {\mathbf {1} _ {K} \mathbf {1} _ {K} ^ {T}}{\sqrt {K + 1} + 1}\right) R ^ {T}} _ {\text {m a t r i x} \mathbb {R} ^ {K} \times (K + 1)} \underbrace {\sqrt {\frac {n}{\Omega_ {j j}}}} _ {\text {s c a l a r}} \underbrace {\left(\frac {1}{n} \sum_ {i = 1} ^ {n} \mathrm {g} _ {i} \mathrm {g} _ {i} ^ {T}\right) ^ {1 / 2}} _ {\text {m a t r i x} \mathbb {R} ^ {(K + 1) \times (K + 1)}} ^ {\dagger} \underbrace {\left(\frac {1}{n} \sum_ {i = 1} ^ {n} \mathrm {V} _ {i} R\right)} _ {\mathbb {R} ^ {(K + 1) \times K}} \underbrace {\hat {A} ^ {T} e _ {j}} _ {\mathbb {R} ^ {K}} \xrightarrow {\mathrm {d}} N (\mathbf {0} _ {K}, I _ {K}) \tag {2.2}
$$
where  $\mathbf{g}_i$  is defined in (1.7) and  $\nabla_{i}$  is defined in Theorem 2.1. Furthermore, for the same  $j\in [p]$
$$
\mathcal {T} _ {n} ^ {j} (X, Y) \stackrel {{d e f}} {{:=}} \frac {n}{\Omega_ {j j}} \left\| \left(\left(\frac {1}{n} \sum_ {i = 1} ^ {n} \mathrm {g} _ {i} \mathrm {g} _ {i} ^ {T}\right) ^ {1 / 2}\right) ^ {\dagger} \left(\frac {1}{n} \sum_ {i = 1} ^ {n} \mathrm {V} _ {i}\right) R \hat {A} ^ {T} e _ {j} \right\| ^ {2} s a t i s f i e s \mathcal {T} _ {n} ^ {j} (X, Y) \xrightarrow {\mathrm {d}} \chi_ {K} ^ {2}. \tag {2.3}
$$
Note that Equations (2.2) and (2.3) is stated using  $\Omega_{jj} = e_j^T\Sigma^{-1}e_j$ . When  $\Sigma$  is unknown, the quantity  $\Omega_{jj}$  in above results can be replaced by its consistent estimate  $\hat{\Omega}_{jj}$  defined in (2.5), and the convergence in distribution results still hold.
Theorem 2.2 is proved in Supplementary Section S4. To the best of our knowledge, Theorem 2.2 is the first result that characterizes the distribution of null MLE coordinate  $\hat{A}^T e_j$  in high-dimensional multinomial logistic regression with 3 or more classes. It is worth mentioning that the quantities  $(\mathsf{g}_i,\mathsf{V}_i,\hat{A})$  used in Theorem 2.2 can be readily computed from the data  $(X,Y)$ . Therefore, Theorem 2.2 lets us test the significance of a specific feature: for testing  $H_0$ , this theorem suggests the test statistic  $\mathcal{T}_n^j (X,Y)$  in (2.3) and the rejection region  $\mathcal{E}_{\alpha}^{j}\stackrel {\mathrm{def}}{:=}\left\{(X,Y):\mathcal{T}_{n}^{j}(X,Y)\geq \chi_{K}^{2}(\alpha)\right\}$ . Under the null hypothesis  $H_0$  in (1.10), Theorem 2.2 guarantees  $\mathbb{P}\big((X,Y)\in \mathcal{E}_{\alpha}^{j}\big)\to \alpha$ . In other words, the test that rejects  $H_0$  if  $(X,Y)\in \mathcal{E}_{\alpha}^{j}$  has type I error converging to  $\alpha$ . The p-value of this test is
$$
\mathrm {p} - \text {v a l u e} = \int_ {\mathcal {T} _ {n} ^ {j} (X, Y)} ^ {+ \infty} f _ {\chi_ {K} ^ {2}} (t) d t, \tag {2.4}
$$
Unknown  $\Omega_{jj} = e_j^T\Sigma^{-1}e_j$ . If  $\Sigma$  is unknown, we describe a consistent estimate of the quantity  $\Omega_{jj}$  appearing in (2.1), (2.2), and (2.3). Under the Gaussian Assumption 2.1, the quantity  $\Omega_{jj}$  is the
(a)  $(n,K) = (4000,2)$
(b)  $(n,K) = (5000,3)$
(c)  $(n,K) = (6000,4)$
Figure 2: Q-Q plots of the test statistic in the left-hand side of (1.13) (in orange) and in the left-hand side of (2.3) (in blue) for different  $(n, K)$  and  $p = 1000$ .
reciprocal of the conditional variance  $\mathrm{Var}(x_{ij}|x_{i, - j})$ , which is also the noise variance in the linear model of regressing  $Xe_{j}$  onto  $X_{-j}$  (the submatrix of  $X$  excluding the  $j$ -th column). According to standard results in linear models, we have  $\Omega_{jj}\| [I_n - X_{-j}(X_{-j}^T X_{-j})^{-1}X_{-j}^T ]Xe_j\| ^2\sim \chi_{n - p + 1}^2$ . Since  $\chi_{n - p + 1}^2 /(n - p + 1)\to 1$  almost surely by the strong law of large numbers,
$$
\hat {\Omega} _ {j j} = (n - p + 1) / \left\| \left[ I _ {n} - X _ {- j} \left(X _ {- j} ^ {T} X _ {- j}\right) ^ {- 1} X _ {- j} ^ {T} \right] X e _ {j} \right\| ^ {2} \tag {2.5}
$$
is a consistent estimator of  $\Omega_{jj}$ . Therefore, the previous asymptotic results in Theorems 2.1 and 2.2 still hold by Slutsky's theorem if we replace  $\Omega_{jj}$  by the estimate  $\hat{\Omega}_{jj}$  in (2.5).
This section presents simulations and a real data analysis to examine finite sample properties of the above results and methods. The source code for generating all of the experimental results in this paper can be found in the supplementary material.
Simulation settings. We set  $p = 1000$  and consider different combinations of  $(n, K)$ . The covariance matrix  $\Sigma$  is specified to be the correlation matrix of an AR(1) model with parameter  $\rho = 0.5$ , that is,  $\Sigma = (0.5^{|i - j|})_{p \times p}$ . We generate the regression coefficients  $A^{*} \in \mathbb{R}^{p \times K}$  once and for all as follows: sample  $A_0 \in \mathbb{R}^{p \times K}$  with first  $\lceil p / 4 \rceil$  rows being i.i.d.  $N(\mathbf{0}, I_K)$ , and set the remaining rows to 0. We then scale the coefficients by defining  $A^{*} = A_{0}(A_{0}^{T}\Sigma A_{0})^{-1 / 2}$  so that  $A^{*T}\Sigma A^{*} = I_{K}$ . With this construction, the  $p$ -th variable is always a null covariate, and we use this null coordinate  $j = p$  to demonstrate the effectiveness of our theoretical results presented in Theorem 2.2 and the suggested test for testing  $H_{0}$  as described in (1.10). Using the above settings, we generate the design matrix  $X \in \mathbb{R}^{n \times p}$  from  $N(0, \Sigma)$ , and then simulate the labels from a multinomial logistic model as given in (1.8), using the coefficients  $A^{*} \in \mathbb{R}^{p \times K}$ . For each simulation setting, we perform 5,000 repetitions.
Assessment of  $\chi^2$  approximations. To assess the  $\chi^2$  approximation (2.3) from this paper and that of the classical theory (1.13), we compute the two  $\chi_K^2$  test statistics for each sample  $(x_i, y_i)_{i=1}^n$ . Figure 2 shows the empirical quantiles of the two statistics versus the  $\chi_K^2$  distribution quantiles. The results demonstrate that the quantiles (in blue) from our high-dimensional theory closely match the 45-degree line (in red), whereas the quantiles (in orange) from the classical theory significantly deviate from the 45-degree line. These findings highlight the accuracy of our proposed  $\chi^2$  approximation (2.3) over the classical result (1.13) when  $p$  is not sufficiently small compared to  $n$ .
Uniformity of null p-values. Recall that the p-value from the classical test (1.13) is given by (1.14), while the p-value from this paper taking into account high-dimensionality is given by (2.4). Figure 3 displays the histograms of these two sets of p-values out of 5000 repetitions. The results in Figure 3 show that the p-values obtained from the classical test deviate significantly from the uniform distribution, with a severe inflation in the lower tail. This indicates that the classical test tends to produce large type I errors due to the excess of p-values close to 0. In contrast, the p-values proposed in this paper exhibit a uniform distribution, further confirming the effectiveness and applicability of the theory in Theorem 2.2 for controlling type I error when testing for null covariates with (1.10).
(c)  $(n,K) = (6000,4)$
Figure 3: Histogram for p-values of the classical test (1.14) (in orange) and of the proposed test (2.4) (in blue) under  $H_0$  in simulated data with different  $(n, K)$  and  $p = 1000$ .
Unknown  $\Omega_{jj}$ . In the situation where the covariance matrix  $\Sigma$  is unknown, we can estimate the diagonal element  $\Omega_{jj} = e_j^T\Sigma^{-1}e_j$  by  $\hat{\Omega}_{jj}$  defined in (2.5). To evaluate the accuracy of the normal and chi-square approximations and the associated test with  $\Omega_{jj}$  replaced by  $\hat{\Omega}_{jj}$ , we conduct simulations similar to those in Figures 2 and 3, but we replace  $\Omega_{jj}$  with its estimate  $\hat{\Omega}_{jj}$ . The results are presented in Figure S1. The plots are visually indistinguishable from the plots using  $\Omega_{jj}$ . These confirm that the chi-square approximation and the associated test using  $\hat{\Omega}_{jj}$  are accurate.
Non-Gaussian covariates and unknown  $\Omega_{jj}$ . Although our theory assumes Gaussian covariates, we expect that the same results hold for other distributions with sufficiently light tails. To illustrate this point, we consider the following two types of non-Gaussian covariates: (i) The design matrix  $X$  has i.i.d. Rademacher entries, i.e.,  $\mathbb{P}(x_{ij} = \pm 1) = \frac{1}{2}$ , (ii) Each  $x_{ij}$  takes on values 0, 1 and 2 with respectively probabilities  $a_j^2$ ,  $2a_j(1 - a_j)$ , and  $(1 - a_j)^2$ , where  $a_j$  varies in [0.25, 0.75]. Each columns of  $X$  are then centered and normalized to have 0 mean and unit variance. This generation of non-Gaussian covariates is adopted from single-nucleotide poly-morphisms (SNPs) example in Sur and Candès [2019]. For these two types of non-Gaussian covariates, we further rescale the feature vectors to ensure that  $x_i$  has the same covariance as in the Gaussian case at the beginning of Section 3, that is  $\Sigma = (0.5^{|i - j|})_{p\times p}$ . We present the Q-Q plots in Figure S2 using the same settings as in Figure S1, with the only difference being that the covariates in Figure S2 are non-Gaussian distributed. The Q-Q plots of  $\mathcal{T}_n^j(X,Y)$  in (2.3) plotted in Figure S2 still closely match the diagonal line. These empirical successes suggest that the normal and  $\chi_K^2$  approximations (2.1)-(2.3) apply to a wider range of covariate distributions beyond normally distributed data.
Real data example. We conduct a real data analysis by applying the proposed test to heart disease data from the UCI Machine Learning Repository (link: http://archive.ics.uci.edu/ml/machine-learning-databases/heart-disease/processed.cleveland.data). After standard data-cleaning processes, the dataset has 297 instances with 13 features, including age, sex, and other attributes. The response variable was transformed into 3 classes (0, 1, and 2) after converting the labels 3 and 4 to 2. To demonstrate the validity of the proposed significance test, we generate a noise variable from a standard normal distribution, resulting in a dataset with 297 instances and 14 variables. We test the significance of this noise variable using both the proposed chi-square test and the classical test, repeating the experiment 10,000 times. The type I error of our proposed test is 0.0508, aligning well with the desired type I error of 0.05. In contrast, the classical test exhibits a type I error of 0.0734, significantly exceeding the desired rate of 0.05. These results confirm the validity of our proposed test on real data corrupted with fake covariate, while the classical test concludes that the fake covariate is significant in more than  $7\%$  of experiments, leading to false discoveries exceeding the desired 0.05 type I error.
4 Discussion and future work
Multinomial logistic regression estimates and their p-values are ubiquitous throughout the sciences for analyzing the significance of explanatory variables on multiclass responses. Following the seminal work of Sur and Candès [2019] in binary logistic regression, this paper develops the first valid tests and p-values for multinomial logistic estimates when  $p$  and  $n$  are of the same order. For 3 or more
classes, this methodology and the corresponding asymptotic normality results in Theorems 2.1 and 2.2 are novel and provide new understanding of multinomial logistic estimates (also known as cross-entropy minimizers) in high-dimensions. We expect similar asymptotic normality and chi-square results to be within reach for loss functions different than the cross-entropy or a different model for the response  $y_{i}$ ; for instance Section S1 provides an extension to the  $q$ -repeated measurements model, where  $q$  responses are observed for each feature vector  $x_{i}$ .
Let us point a few follow-up research directions that we leave open for future work. A first open problem regards extensions of our methodology to confidence sets for  $e_j^T \mathsf{B}^*$  when  $H_0$  in (1.10) is violated for the  $j$ -th covariate. This would require more stringent assumptions on the generative model than Assumption 2.1 as  $\mathsf{B}^*$  there is not identifiable (e.g., modification of both  $\mathsf{B}^*$  and  $f(\cdot, \cdot)$  in Assumption 2.1 is possible without changing  $y_i$ ). A second open problem is to relate this paper's theory to the fixed-point equations and limiting Gaussian model obtained in multiclass models, e.g., Loureiro et al. [2021]. While it may be straightforward to obtain the limit of  $\frac{1}{n} \sum_{i=1}^{n} \mathsf{g}_i \mathsf{g}_i^T$  and of the empirical distribution of the rows of  $\hat{A}$  in this context (e.g., using Corollary 3 in Loureiro et al. [2021]), the relationship between the fixed-point equations and the matrix  $\frac{1}{n} \sum_{i=1}^{n} \mathsf{V}_i$  appearing in (2.1) is unclear and not explained by typical results from this literature. A third open problem is to characterize the exact phase transition below which the multinomial logistic MLE exists and is bounded with high-probability (Assumption 2.4); while this is settled for two classes [Candes and Sur, 2020] and preliminary results are available for 3 or more classes [Loureiro et al., 2021, Kini and Thrampoulidis, 2021], a complete understanding of this phase transition is currently lacking. A last interesting open problem is to prove that our theory extends to non-Gaussian data, as observed in simulations. This challenging problem is often referred to as "universality" and has received intense attention recently [Gerace et al., 2022, Han and Shen, 2022, Montanari and Saeed, 2022, Pesce et al., 2023, Dandi et al., 2023], showing that in several settings of interest (although none exactly matching the one considered in the present paper), the asymptotic behavior of the minimizers is unchanged if the distribution of the covariates is modified from normal to another distribution with the same covariance.
P.C. Bellec's research was partially supported by the NSF Grant DMS-1945428. The authors acknowledge the Office of Advanced Research Computing (OARC) at Rutgers, The State University of New Jersey for providing access to the Amarel cluster and associated research computing resources that have contributed to the results reported here. URL: https://oarc.rutgers.edu.