SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Kai Tan
kai.tan@rutgers.edu
Pierre C. Bellec
pierre.bellec@rutgers.edu
This paper investigates the asymptotic distribution of the maximum-likelihood estimate (MLE) in multinomial logistic models in the high-dimensional regime where dimension and sample size are of the same order. While classical large-sample theory provides asymptotic normality of the MLE under certain conditions, such classical results are expected to fail in high-dimensions as documented for the binary logistic case in the seminal work of Sur and Candès [2019]. We address this issue in classification problems with 3 or more classes, by developing asymptotic normality and asymptotic chi-square results for the multinomial logistic MLE (also known as cross-entropy minimizer) on null covariates. Our theory leads to a new methodology to test the significance of a given feature. Extensive simulation studies on synthetic data corroborate these asymptotic results and confirm the validity of proposed p-values for testing the significance of a given feature.
Multinomial logistic modeling has become a cornerstone of classification problems in machine learning, as witnessed by the omnipresence of both the cross-entropy loss (multinomial logistic loss) and the softmax function (gradient of the multinomial logistic loss) in both applied and theoretical machine learning. We refer to Cramer [2002] for an account of the history and early developments of logistic modeling.
Throughout, we consider a classification problem with  $K + 1$  possible labels where  $K$  is a fixed constant. This paper tackles asymptotic distributions of multinomial logistic estimates (or cross-entropy minimizers) in generalized linear models with moderately high-dimensions, where sample size  $n$  and dimension  $p$  have the same order, for instance  $n, p \to +\infty$  simultaneously while the ratio  $p / n$  converges to a finite constant. Throughout the paper, let  $[n] = \{1,2,\dots,n\}$  for all  $n \in \mathbb{N}$ , and  $I\{\text{statement}\}$  be the 0-1 valued indicator function, equal to 1 if statement is true and 0 otherwise (e.g.,  $I\{y_{i} = 1\}$  in the next paragraph equals 1 if  $y_{i} = 1$  holds and 0 otherwise).
The case of binary logistic regression. Let  $\rho(t) = \log(1 + e^t)$  be the logistic loss and  $\rho'(t) = 1/(1 + e^{-t})$  be its derivative, often referred to as the sigmoid function. In the current moderately-high dimensional regime where  $n, p \to +\infty$  with  $p/n \to \kappa > 0$  for some constant  $\kappa$ , recent works [Candès and Sur, 2020, Sur and Candès, 2019, Zhao et al., 2022] provide a detailed theoretical understanding of the behavior of the logistic Maximum Likelihood Estimate (MLE) in binary logistic regression models. Observing independent observations  $(x_i, y_i)_{i \in [n]}$  from a logistic model defined as  $\mathbb{P}(y_i = 1|x_i) = \rho'(x_i^T\beta)$  where  $x_i \sim N(\mathbf{0}, n^{-1}I_p)$ , and  $\lim_{n \to \infty} \| \beta \|^2 / n = \gamma^2$  for a constant  $\gamma$  for the limiting squared norm of the unknown regression vector  $\beta$ . These works prove that the behavior of the MLE  $\hat{\beta} = \arg \min_{b \in \mathbb{R}^p} \sum_{i=1}^{n} \rho(x_i^T b) - I\{y_i = 1\} x_i^T b$  is summarized by the solution
$(\alpha_{*},\sigma^{*},\lambda_{*})$  of the system of three equations
$$
\left\{\begin{array}{l l}\sigma^ {2}&= \frac {1}{\kappa^ {2}} \mathbb {E} \left[ 2 \rho^ {\prime} \left(\gamma Z _ {1}\right) \left(\lambda \rho^ {\prime} \left(\operatorname {p r o x} _ {\lambda \rho} (- \alpha \gamma Z _ {1} + \sqrt {\kappa} \sigma Z _ {2}))\right) ^ {2} \right] \right.\\0&= \mathbb {E} \left[ \rho^ {\prime} \left(\gamma Z _ {1}\right) \lambda \rho^ {\prime} \left(\operatorname {p r o x} _ {\lambda \rho} (- \alpha \gamma Z _ {1} + \sqrt {\kappa} \sigma Z _ {2}))\right) \right]\\1 - \kappa&= \mathbb {E} \left[ \right. 2 \rho^ {\prime} \left(\gamma Z _ {1}\right) / \left( \right.1 + \lambda \rho^ {\prime \prime} \left(\operatorname {p r o x} _ {\lambda \rho} (- \alpha \gamma Z _ {1} + \sqrt {\kappa} \sigma Z _ {2}))\right)\left. \right]\end{array}, \right. \tag {1.1}
$$
where  $(Z_{1}, Z_{2})$  are i.i.d.  $N(0,1)$  random variables and the proximal operator is defined as  $\mathrm{prox}_{\lambda \rho}(z) = \arg \min_{t \in \mathbb{R}} \left\{\lambda \rho(t) + (t - z)^{2} / 2\right\}$ . The system (1.1) characterizes, among others, the following behavior of the MLE  $\hat{\beta}$ : for almost any  $(\gamma, \kappa)$ , the system admits a solution if and only if  $\hat{\beta}$  exists with probability approaching one and in this case,  $\|\hat{\beta}\|^{2} / n$  and  $\|\hat{\beta} - \beta\|^{2} / n$  both have finite limits that may be expressed as simple functions of  $(\alpha_{*}, \sigma_{*}, \lambda_{*})$ , and for any feature  $j \in [p]$  such that  $\beta_{j} = 0$  (i.e.,  $j$  is a null covariate), the  $j$ -th coordinate of the MLE satisfies
$$
\hat {\beta} _ {j} \xrightarrow {\mathrm {d}} N \left(0, \sigma_ {*} ^ {2}\right).
$$
The proofs in Sur and Candès [2019] are based on approximate message passing (AMP) techniques; we refer to Berthier et al. [2020], Feng et al. [2022], Gerbelot and Berthier [2021] and the references therein for recent surveys and general results. More recently, Zhao et al. [2022] extended the result of Sur and Candès [2019] from isotropic design to Gaussian covariates with an arbitrary covariance structure: if now  $x_{i} \sim N(\mathbf{0},\Sigma)$  for some positive definite  $\Sigma$  and  $\lim_{n,p\to +\infty}\beta^T\Sigma \beta = \kappa$ , null covariates  $j \in [p]$  (in the sense that  $y_{i}$  is independent of  $x_{ij}$  given  $(x_{ik})_{k \in [p] \setminus \{j\}}$ ) of the MLE satisfy
$$
\left(n / \Omega_ {j j}\right) ^ {1 / 2} \hat {\beta} _ {j} \xrightarrow {\mathrm {d}} N \left(0, \sigma_ {*} ^ {2}\right), \tag {1.2}
$$
where  $\sigma_{*}$  is the same solution of (1.1) and  $\Omega = \Sigma^{-1}$ . Zhao et al. [2022] also obtained asymptotic normality results for non-null covariates, that is, features  $j\in [p]$  such that  $\beta_{j}\neq 0$ . The previous displays can be used to test the null hypothesis  $H_0:y_i$  is independent of  $x_{ij}$  given  $(x_{ik})_{k\in [p]\setminus \{j\}}$  and develop the corresponding p-values if  $\sigma_{*}$  is known; in this binary logistic regression model the ProbeFrontier [Sur and Candès, 2019] and SLOE Yadowsky et al. [2021] give means to estimate the solutions  $(\alpha_{*},\sigma_{*},\lambda_{*})$  of system (1.1) without the knowledge of  $\gamma$ . Mai et al. [2019] studied the performance of Ridge regularized binary logistic regression in mixture models. Salehi et al. [2019] extended Sur and Candès [2019] to separable penalty functions. Bellec [2022] derived asymptotic normality results similar to (1.2) in single-index models including binary logistic regression without resorting to the system (1.1), showing that for a null covariate  $j\in [p]$  in the unregularized case that
$$
\left(n / \Omega_ {j j}\right) ^ {1 / 2} (\hat {v} / \hat {r}) \hat {\beta} _ {j} \xrightarrow {\mathrm {d}} N (0, 1) \tag {1.3}
$$
where  $\hat{v} = \frac{1}{n}\sum_{i=1}^{n}\rho''(x_i^T\hat{\beta}) - \rho''(x_i^T\hat{\beta})^2 x_i^T[\sum_{l=1}^{n}x_l\rho''(x_l^T\hat{\beta})x_l^T]^{-1}x_i$  is scalar and so is  $\hat{r}^2 = \frac{1}{n}\sum_{i=1}^{n}(I\{y_i = 1\} - \rho'(x_i^T\hat{\beta}))^2$ . In summary, in this high dimensional binary logistic model,
(i) The phase transition from Candès and Sur [2020] splits the  $(\gamma, \kappa)$  plane into two connected components: in one component the MLE does not exist with high probability, in the other component the MLE exists and  $\|\Sigma^{1/2}\hat{\beta}\|^2$  is bounded with high probability (boundedness is a consequence of the fact that  $\|\Sigma^{1/2}\hat{\beta}\|^2$  or  $\|\Sigma^{1/2}(\hat{\beta} - \beta)\|^2$  admits finite limits);
(ii) In the component of the  $(\gamma, \kappa)$  plane where the MLE exists, for any null covariate  $j \in [p]$ , the asymptotic normality results (1.2)-(1.3) hold.
Multiclass classification. The goal of this paper is to develop a theory for the asymptotic normality of the multinomial logistic regression MLE (or cross-entropy minimizer) on null covariates when the number of classes,  $K + 1$ , is greater than 2 and  $n, p$  are of the same order. In other words, we aim to generalize results such as (1.2) or (1.3) for three or more classes. Classification datasets with 3 or more classes are ubiquitous in machine learning (MNIST, CIFAR to name a few), which calls for such multiclass generalizations. In Gaussian mixtures and logistic models, Thrangpoulidis et al. [2020] derived characterizations of the performance of least-squares and class-averaging estimators, excluding cross-entropy minimizers or minimizers of non-linear losses. Loureiro et al. [2021] extended Sur and Candès [2019], Zhao et al. [2022], Salehi et al. [2019] to multiclass classification problems in a Gaussian mixture model, and obtained the fixed-point equations that characterize the performance and empirical distribution of the minimizer of the cross-entropy loss plus a convex regularizer. In the same vein as Loureiro et al. [2021], Cornacchia et al. [2022] studied the limiting
fixed-point equations in a multiclass teacher-student learning model where labels are generated by a noiseless channel with response  $\arg \min_{k\in \{1,\dots ,K\}}x_i^T\beta_k$  where  $\beta_{k}\in \mathbb{R}^{p}$  is unknown for each class  $k$ . These two aforementioned works assume a multiclass Gaussian mixture model, which is different than the normality assumption for  $x_{i}$  used in the present paper. More importantly, these results cannot be readily used for the purpose testing significant covariates (cf. (1.10) below) since solving the fixed-point equations require the knowledge of several unknown parameters, including the limiting spectrum of the mixture covariances and empirical distributions of the mixture means (cf. for instance Corollary 3 in Loureiro et al. [2021]). In the following sections, we fill this gap with a new methodology to test the significance of covariates. This is made possible by developing new asymptotic normality results for cross-entropy minimizers that generalize (1.3), without relying on the low-dimensional fixed-point equations.
Notation. Throughout,  $I_{p} \in \mathbb{R}^{p \times p}$  is the identity matrix, for a matrix  $A \in \mathbb{R}^{m \times n}$ ,  $A^T$  denotes the transpose of  $A$ ,  $A^\dagger$  denotes the Moore-Penrose inverse of  $A$ . If  $A$  is psd,  $A^{1/2}$  denotes the unique symmetric square root, i.e., the unique positive semi-definite matrix such that  $(A^{1/2})^2 = A$ . The symbol  $\otimes$  denotes the Kronecker product of matrices. Given two matrices  $A \in \mathbb{R}^{n \times k}$ ,  $B \in \mathbb{R}^{n \times q}$  with the same number or rows,  $(A, B) \in \mathbb{R}^{n \times (k + q)}$  is the matrix obtained by stacking the columns of  $A$  and  $B$  horizontally. If  $v \in \mathbb{R}^n$  is a column vector with dimension equal to the number of rows in  $A$ , we construct  $(A, v) \in \mathbb{R}^{n \times (k + 1)}$  similarly. We use  $\mathbf{0}_n$  and  $\mathbf{1}_n$  to denote the all-zeros vector and all-ones vector in  $\mathbb{R}^n$ , respectively; we do not bold vectors and matrices other than  $\mathbf{0}_n$  and  $\mathbf{1}_n$ . We may omit the subscript giving the dimension if clear from context; e.g., in  $I_{K + 1} - \frac{\mathbf{11}^T}{K + 1}$  the vector  $\mathbf{1}$  is in  $\mathbb{R}^{K + 1}$ . The Kronecker product between two matrices is denoted by  $\otimes$  and  $\operatorname{vec}(M) \in \mathbb{R}^{nd}$  is the vectorization operator applied to a matrix  $M \in \mathbb{R}^{n \times d}$ . For an integer  $K \geq 2$  and  $\alpha \in (0, 1)$ , the quantile  $\chi_K^2(\alpha)$  is the unique real number satisfying  $\mathbb{P}(W > \chi_K^2(\alpha)) = \alpha$  where  $W$  has a chi-square distribution with  $K$  degrees of freedom. The symbols  $\xrightarrow{\mathrm{d}}$  and  $\xrightarrow{\mathrm{p}}$  denote convergence in distribution and in probability.
Throughout, classical asymptotic regime refers to the scenario where the feature dimension  $p$  is fixed and the sample size  $n$  goes to infinity. In contrast, the term high-dimensional regime refers to the situation where  $n$  and  $p$  both tend to infinity with the ratio  $p / n$  converging to a limit smaller than 1.
1.1 Multinomial logistic regression
Consider a multinomial logistic regression model with  $K + 1$  classes. We have  $n$  i.i.d. data samples  $\{(x_{i},y_{i})\}_{i = 1}^{n}$ , where  $x_{i}\in \mathbb{R}^{p}$  is the feature vector and  $y_{i} = (y_{i1},\dots,y_{i(K + 1)})^{T}\in \mathbb{R}^{K + 1}$  is the response. Each response  $y_{i}$  is the one-hot encoding of a single label, i.e.,  $y_{i}\in \{0,1\}^{K + 1}$  with  $\sum_{k = 1}^{K + 1}y_{ik} = 1$  such that  $y_{ik} = 1$  if and only if the label for  $i$ -th observation is  $k$ . A commonly used generative model for  $y_{i}$  is the multinomial regression model, namely
$$
\mathbb {P} \left(\mathrm {y} _ {i k} = 1 \mid x _ {i}\right) = \frac {\exp \left(x _ {i} ^ {T} \mathrm {B} ^ {*} e _ {k}\right)}{\sum_ {k ^ {\prime} = 1} ^ {K + 1} \exp \left(x _ {i} ^ {T} \mathrm {B} ^ {*} e _ {k ^ {\prime}}\right)}, \quad k \in \{1, 2, \dots , K + 1 \} \tag {1.4}
$$
where  $\mathsf{B}^* \in \mathbb{R}^{p \times (K + 1)}$  is an unknown logistic model parameter and  $e_k \in \mathbb{R}^{K + 1}$ ,  $e_{k'} \in \mathbb{R}^{K + 1}$  are the  $k$ -th and  $k'$ -th canonical basis vectors. The MLE for  $\mathsf{B}^*$  in the model (1.4) is any solution that minimizes the cross-entropy loss,
$$
\hat {\mathsf {B}} \in \arg \min  _ {\mathsf {B} \in \mathbb {R} ^ {p \times (K + 1)}} \sum_ {i = 1} ^ {n} \mathsf {L} _ {i} \left(\mathsf {B} ^ {T} x _ {i}\right), \tag {1.5}
$$
where  $\mathsf{L}_i:\mathbb{R}^{K + 1}\to \mathbb{R}$  is defined as  $\mathsf{L}_i(\mathsf{u}) = -\sum_{k = 1}^{K + 1}\mathsf{y}_{ik}\mathsf{u}_k + \log \sum_{k' = 1}^{K + 1}\exp (\mathsf{u}_{k'})$  . If the solution set in (1.5) is non-empty, we define for each observation  $i\in [n]$  the vector of predicted probabilities  $\hat{\mathfrak{p}}_i = (\hat{\mathfrak{p}}_{i1},\dots,\hat{\mathfrak{p}}_{i(K + 1)})^T$  with
$$
\hat {\mathfrak {p}} _ {i k} \stackrel {\text {d e f}} {=} \mathbb {P} (\hat {\mathcal {Y}} _ {i k} = 1) = \frac {\exp \left(x _ {i} ^ {T} \hat {\mathsf {B}} e _ {k}\right)}{\sum_ {k ^ {\prime} = 1} ^ {K + 1} \exp \left(x _ {i} ^ {T} \hat {\mathsf {B}} e _ {k ^ {\prime}}\right)} \quad \text {f o r e a c h} k \in \{1, \dots , K + 1 \}. \tag {1.6}
$$
Our results will utilize the gradient and Hessian of  $\mathsf{L}_i$  evaluated at  $\hat{\mathbf{B}}^T x_i$ , denoted by
$$
\mathbf {g} _ {i} \stackrel {\text {d e f}} {=} \nabla \mathsf {L} _ {i} (\hat {\mathsf {B}} ^ {T} x _ {i}) = - \mathbf {y} _ {i} + \hat {\mathbf {p}} _ {i}, \quad \mathsf {H} _ {i} \stackrel {\text {d e f}} {=} \nabla^ {2} \mathsf {L} _ {i} (\hat {\mathsf {B}} ^ {T} x _ {i}) = \operatorname {d i a g} (\hat {\mathbf {p}} _ {i}) - \hat {\mathbf {p}} _ {i} \hat {\mathbf {p}} _ {i} ^ {T}. \tag {1.7}
$$
The quantities  $(\hat{\mathsf{B}},\hat{\mathsf{p}}_i,\mathsf{g}_i,\mathsf{H}_i)$  can be readily computed from the data  $\{(x_i,y_i)\}_{i = 1}^n$  . To be specific, the MLE  $\hat{\mathsf{B}}$  in (1.5) can be obtained by invoking a multinomial regression solver (e.g., sklearn.linear_model.LogisticRegression from Pedregosa et al. [2011]), and the quantities  $\hat{\mathsf{p}}_i,\mathsf{g}_i,\mathsf{H}_i$  can be further computed from eqs. (1.6) and (1.7) by a few matrix multiplications and application of the softmax function.
Log-odds model and reference class. The matrix  $\mathsf{B}^*$  in (1.4) is not identifiable since the conditional distribution of  $y_{i}|x_{i}$  in the model (1.4) remains unchanged if we replace columns of  $\mathsf{B}^*$  by  $\mathsf{B}^* - b\mathbf{1}_{K + 1}^T$  for any  $b \in \mathbb{R}^p$ . In order to obtain an identifiable model, a classical and natural remedy is to model the log-odds, here with the class  $K + 1$  as the reference class:
$$
\log \frac {\mathbb {P} \left(\mathrm {y} _ {i k} = 1 \mid x _ {i}\right)}{\mathbb {P} \left(\mathrm {y} _ {i (K + 1)} = 1 \mid x _ {i}\right)} = x _ {i} ^ {T} A ^ {*} e _ {k}, \quad \forall k \in [ K ] \tag {1.8}
$$
where  $e_k$  is the  $k$ -th canonical basis vector of  $\mathbb{R}^K$ , and  $A^* \in \mathbb{R}^{p \times K}$  is the unknown parameter. The matrix  $A^* \in \mathbb{R}^{p \times K}$  in log-odds model (1.8) is related to  $\mathsf{B}^* \in \mathbb{R}^{p \times (K + 1)}$  in the model (1.4) by  $A^* = \mathsf{B}^*(I_K, -\mathbf{1}_K)^T$ . This log-odds model has two benefits: First it is identifiable since the unknown matrix  $A^*$  is uniquely defined. Second, the matrix  $A^*$  lends itself well to interpretation as its  $k$ -th column represents the contrast coefficient between class  $k$  and the reference class  $K + 1$ .
The MLE  $\hat{A}$  of  $A^{*}$  in (1.8) is  $\hat{A} = \arg \min_{A\in \mathbb{R}^{p\times K}}\sum_{i = 1}^{n}\mathsf{L}_{i}((A,\mathbf{0}_{p})^{T}x_{i})$ . If the solution set in (1.5) is non-empty,  $\hat{A}$  is related to any solution  $\hat{\mathsf{B}}$  in (1.5) by  $\hat{A} = \hat{\mathsf{B}} (I_K, - \mathbf{1}_K)^T$ . Equivalently,
$$
\hat {A} _ {j k} = \hat {\mathrm {B}} _ {j k} - \hat {\mathrm {B}} _ {j (K + 1)} \tag {1.9}
$$
for each  $j \in [p]$  and  $k \in [K]$ . If there are three classes (i.e.  $K + 1 = 3$ ), this parametrization allows us to draw scatter plots of realizations of  $\sqrt{n} e_j^T\hat{A} = (\sqrt{n}\hat{A}_{j1},\sqrt{n}\hat{A}_{j,2})$  as in Figure 1.
1.2 Hypothesis testing for the  $j$ -th feature and classical asymptotic normality for MLE
Hypothesis testing for the  $j$ -th feature. Our goal is to develop a methodology to test the significance of the  $j$ -th feature. Specifically, for a desired confidence level  $(1 - \alpha) \in (0, 1)$  (say,  $1 - \alpha = 0.95$ ) and a given feature  $j \in [p]$  of interest, our goal is to test
$$
H _ {0}: y _ {i} \text {i s c o n d i t i o n a l y i n d e p e n d e n t o f} x _ {i j} \text {g i v e n} \left(x _ {i j ^ {\prime}}\right) _ {j ^ {\prime} \in [ p ] \backslash \{j \}}. \tag {1.10}
$$
Namely, we want to test whether the  $j$ -th variable is independent from the response given all other explanatory variables  $(x_{ij'}, j' \in [p] \setminus \{j\})$ . Assuming normally distributed  $x_i$  and a multinomial model as in (1.4) or (1.8), it is equivalent to test
$$
H _ {0}: e _ {j} ^ {T} A ^ {*} = \mathbf {0} _ {K} ^ {T} \quad \text {v e r s u s} \quad H _ {1}: e _ {j} ^ {T} A ^ {*} \neq \mathbf {0} _ {K} ^ {T}, \tag {1.11}
$$
where  $e_j \in \mathbb{R}^p$  is the  $j$ -th canonical basis vector.
If the MLE  $\hat{\mathbf{B}}$  in (1.5) exists in the sense that the solution set in (1.5) is nonempty, the conjecture that rejecting  $H_0$  when  $e_j^T\hat{\mathbf{B}}$  is far from  $\mathbf{0}_{K + 1}$  is a reasonable starting point. The important question, then, is to determine a quantitative statement for the informal "far from  $\mathbf{0}_{K + 1}$ ", similarly to (1.2) or (1.3) in binary logistic regression.
Classical theory with  $p$  fixed. If  $p$  is fixed and  $n \to \infty$  in model (1.8), classical maximum likelihood theory [Van der Vaart, 1998, Chapter 5] provides the asymptotic distribution of the MLE  $\hat{A}$ , which can be further used to test (1.11). Briefly, if  $x$  has the same distribution as any  $x_{i}$ , the MLE  $\hat{A}$  in the multinomial logistic model is asymptotically normal with
$$
\sqrt {n} \left(\operatorname {v e c} (\hat {A}) - \operatorname {v e c} \left(A ^ {*}\right)\right) \xrightarrow {\mathrm {d}} N \left(\mathbf {0}, \mathcal {I} ^ {- 1}\right) \quad \text {w h e r e} \quad \mathcal {I} = \mathbb {E} \left[ \left(x x ^ {T}\right) \otimes \left(\operatorname {d i a g} \left(\pi^ {*}\right) - \pi^ {*} \pi^ {* T}\right) \right]
$$
is the Fisher information matrix evaluated at the true parameter  $A^{*}$ ,  $\operatorname{vec}(\cdot)$  is the usual vectorization operator, and  $\pi^{*} \in \mathbb{R}^{K}$  has random entries  $\pi_{k}^{*} = \exp (x^{T}A^{*}e_{k}) / (1 + \sum_{k^{\prime} = 1}^{K}\exp (x^{T}A^{*}e_{k^{\prime}}))$  for each  $k \in [K]$ . In particular, under  $H_0: e_j^T A^* = \mathbf{0}_K^T$ ,
$$
\sqrt {n} \hat {A} ^ {T} e _ {j} \xrightarrow {\mathrm {d}} N (\mathbf {0}, S _ {j}) \tag {1.12}
$$
where  $S_{j} = (e_{j}^{T}\otimes I_{K})\mathcal{I}^{-1}(e_{j}\otimes I_{K}) = e_{j}^{T}(\operatorname {cov}(x))^{-1}e_{j}[\mathbb{E}(\operatorname {diag}(\pi^{*}) - \pi^{*}\pi^{*T})]^{-1}$ . When (1.12) holds, by the delta method we also have  $\sqrt{n} S_j^{-1 / 2}\hat{A}^T e_j\xrightarrow{\mathrm{d}}N(\mathbf{0},I_K)$  and
$$
n \| S _ {j} ^ {- 1 / 2} \hat {A} ^ {T} e _ {j} \| ^ {2} \xrightarrow {\mathrm {d}} \chi_ {K} ^ {2}. \tag {1.13}
$$
where the limiting distribution is chi-square with  $K$  degrees of freedom. This further suggests the size  $\alpha$  test that rejects  $H_0$  when  $T_{n}^{j}(X,Y) > \chi_{K}^{2}(\alpha)$ , where  $T_{n}^{j}(X,Y) = n\| S_{j}^{-1 / 2}\hat{A}^{T}e_{j}\|^{2}$  is the test statistic. If (1.13) holds, this test is guaranteed to have a type I error converging to  $\alpha$ . The p-value of this test is given by
$$
\int_ {T _ {n} ^ {j} (X, Y)} ^ {+ \infty} f _ {\chi_ {K} ^ {2}} (t) d t, \tag {1.14}
$$
where  $f_{\chi_K^2}(\cdot)$  is the density of the chi-square distribution with  $K$  degrees of freedom.
As discussed in the introduction, Sur and Candès [2019] showed that in binary logistic regression, classical normality results for the MLE such as (1.12) fail in the high-dimensional regime because the variance in (1.12) underestimates the variability of the MLE even for null covariates; see also the discussion surrounding (1.2). Our goal is to develop, for classification problems with  $K + 1 \geq 3$  classes, a theory that correctly characterizes the asymptotic distribution of  $\hat{A}^T e_j$  for a null covariate  $j \in [p]$  in the high-dimensional regime.
We present first some motivating simulations that demonstrate the failure of classical normal approximation (1.12) in finite samples. These simulations are conducted for various configurations of  $(n,p)$  with  $K + 1 = 3$  classes. We fix the true parameter  $A^{*}$  and obtain 1000 realizations of  $(\hat{A}_{j1},\hat{A}_{j2})$  by independently resampling the data  $\{(x_i,y_i)\}_{i = 1}^n$  1000 times. If the result (1.12) holds, then  $\mathbb{P}(\sqrt{n}\hat{A}^T e_j\in \mathcal{C}_\alpha^j)\to 1 - \alpha$ , where  $\mathcal{C}_\alpha^j = \{u\in \mathbb{R}^K:\| S_j^{-1 / 2}u\| \leq \chi_K^2 (\alpha)\}$ . Figure 1 displays scatter plots of  $\sqrt{n} (\hat{A}_{j1},\hat{A}_{j2})$  along with the boundary of  $95\%$  confidence set  $\mathcal{C}_\alpha^j$  with  $\alpha = 0.05$ . We observe that, across the three different configurations of  $(n,p)$ , the  $95\%$  confidence sets from our theory (Theorem 2.2 presented in next section) cover around  $95\%$  of the realizations, while the set  $\mathcal{C}_\alpha^j$  from classical theory only covers approximately  $30\%$  of the points, which is significantly lower than the desired coverage rate of  $95\%$ . Intuitively and by analogy with results in binary classification [Sur and Candès, 2019], this is because the classical theory (1.12) underestimates the variation of the MLE in the high-dimensional regime. Motivated by this failure of classical MLE theory and the results in binary classification [Sur and Candès, 2019, among others], the goal of this paper is to develop a theory for multinomial logistic regression that achieves the following objectives:
(a)  $(n,p) = (2000,600)$
(b)  $(n,p) = (3500,1000)$