Title: Multinomial Logistic Regression: Asymptotic Normality on Null Covariates in High-Dimensions
Venue: NeurIPS 2023
Accepted: yes
Keywords: High-dimensional statistics

statistical inference

multi-class classification

asymptotic normality

multinomial logistic regression
Abstract: This paper investigates the asymptotic distribution of the maximum-likelihood estimate (MLE) in multinomial logistic models in the high-dimensional regime where dimension and sample size are of the same order. While classical large-sample theory provides asymptotic normality of the MLE under certain conditions, such classical results are expected to fail in high-dimensions as documented for the binary logistic case in the seminal work of Sur and Candès [2019]. We address this issue in classification problems with 3 or more classes, by developing asymptotic normality and asymptotic chi-square results for the multinomial logistic MLE (also known as cross-entropy minimizer) on null covariates. Our theory leads to a new methodology to test the significance of a given feature. Extensive simulation studies on synthetic data corroborate these asymptotic results and confirm the validity of proposed p-values for testing the significance of a given feature.

Multinomial Logistic Regression: Asymptotic Normality on Null Covariates in High-Dimensions

Abstract

This paper investigates the asymptotic distribution of the maximum-likelihood estimate (MLE) in multinomial logistic models in the high-dimensional regime where dimension and sample size are of the same order. While classical largesample theory provides asymptotic normality of the MLE under certain conditions, such classical results are expected to fail in high-dimensions as documented for the binary logistic case in the seminal work of Sur and Candès [2019]. We address this issue in classification problems with 3 or more classes, by developing asymptotic normality and asymptotic chi-square results for the multinomial logistic MLE (also known as cross-entropy minimizer) on null covariates. Our theory leads to a new methodology to test the significance of a given feature. Extensive simulation studies on synthetic data corroborate these asymptotic results and confirm the validity of proposed p-values for testing the significance of a given feature.

1 Introduction

Multinomial logistic modeling has become a cornerstone of classification problems in machine learning, as witnessed by the omnipresence of both the cross-entropy loss (multinomial logistic loss) and the softmax function (gradient of the multinomial logistic loss) in both applied and theoretical machine learning. We refer to Cramer [2002] for an account of the history and early developments of logistic modeling.

Throughout, we consider a classification problem with K +1 possible labels where K is a fixed constant. This paper tackles asymptotic distributions of multinomial logistic estimates (or crossentropy minimizers) in generalized linear models with moderately high-dimensions, where sample size n and dimension p have the same order, for instance n, p → + ∞ simultaneously while the ratio p/n converges to a finite constant. Throughout the paper, let [ n ] = { 1 , 2 , . . . , n } for all n ∈ N , and I { statement } be the 0-1 valued indicator function, equal to 1 if statement is true and 0 otherwise (e.g., I { y i = 1 } in the next paragraph equals 1 if y i = 1 holds and 0 otherwise).

The case of binary logistic regression. Let ρ ( t ) = log(1 + e t ) be the logistic loss and ρ ′ ( t ) = 1 / (1+ e -t ) be its derivative, often referred to as the sigmoid function. In the current moderately-high dimensional regime where n, p → + ∞ with p/n → κ > 0 for some constant κ , recent works [Candès and Sur, 2020, Sur and Candès, 2019, Zhao et al., 2022] provide a detailed theoretical understanding of the behavior of the logistic Maximum Likelihood Estimate (MLE) in binary logistic regression models. Observing independent observations ( x i , y i ) i ∈ [ n ] from a logistic model defined as P ( y i = 1 | x i ) = ρ ′ ( x T i β ) where x i ∼ N ( 0 , n -1 I p ) , and lim n →∞ ∥ β ∥ 2 /n = γ 2 for a constant γ for the limiting squared norm of the unknown regression vector β . These works prove that the behavior of the MLE ˆ β = arg min b ∈ R p ∑ n i =1 ρ ( x T i b ) -I { y i = 1 } x T i b is summarized by the solution

( α ∗ , σ ∗ , λ ∗ ) of the system of three equations

where ( Z 1 , Z 2 ) are i.i.d. N (0 , 1) random variables and the proximal operator is defined as prox λρ ( z ) = arg min t ∈ R { λρ ( t ) + ( t -z ) 2 / 2 } . The system (1.1) characterize, among others, the following behavior of the MLE ˆ β : for almost any ( γ, κ ) , the system admits a solution if and only if ˆ β exists with probability approaching one and in this case, ∥ ˆ β ∥ 2 /n and ∥ ˆ β -β ∥ 2 /n both have finite limits that may be expressed as simple functions of ( α ∗ , σ ∗ , λ ∗ ) , and for any feature j ∈ [ p ] such that β j = 0 (i.e., j is a null covariate), the j -th coordinate of the MLE satisfies

The proofs in Sur and Candès [2019] are based on approximate message passing (AMP) techniques; we refer to Berthier et al. [2020], Feng et al. [2022], Gerbelot and Berthier [2021] and the references therein for recent surveys and general results. More recently, Zhao et al. [2022] extended the result of Sur and Candès [2019] from isotropic design to Gaussian covariates with an arbitrary covariance structure: if now x i ∼ N ( 0 , Σ) for some positive definite Σ and lim n,p → + ∞ β T Σ β = κ , null covariates j ∈ [ p ] (in the sense that y i is independent of x ij given ( x ik ) k ∈ [ p ] \{ j } ) of the MLE satisfy

̸

where σ ∗ is the same solution of (1.1) and Ω = Σ -1 . Zhao et al. [2022] also obtained asymptotic normality results for non-null covariates, that is, features j ∈ [ p ] such that β j = 0 . The previous displays can be used to test the null hypothesis H 0 : y i is independent of x ij given ( x ik ) k ∈ [ p ] \{ j } and develop the corresponding p-values if σ ∗ is known; in this binary logistic regression model the ProbeFrontier [Sur and Candès, 2019] and SLOE Yadlowsky et al. [2021] give means to estimate the solutions ( α ∗ , σ ∗ , λ ∗ ) of system (1.1) without the knowledge of γ . Mai et al. [2019] studied the performance of Ridge regularized binary logistic regression in mixture models. Salehi et al. [2019] extended Sur and Candès [2019] to separable penalty functions. Bellec [2022] derived asymptotic normality results similar to (1.2) in single-index models including binary logistic regression without resorting to the system (1.1), showing that for a null covariate j ∈ [ p ] in the unregularized case that

where ˆ v = 1 n ∑ n i =1 ρ ′′ ( x T i ˆ β ) -ρ ′′ ( x T i ˆ β ) 2 x T i [ ∑ n l =1 x l ρ ′′ ( x T l ˆ β ) x T l ] -1 x i is scalar and so is ˆ r 2 = 1 n ∑ n i =1 ( I { y i = 1 } -ρ ′ ( x T i ˆ β )) 2 . In summary, in this high dimensional binary logistic model,

(i) The phase transition from Candès and Sur [2020] splits the ( γ, κ ) plane into two connected components: in one component the MLE does not exist with high probability, in the other component the MLE exists and ∥ Σ 1 / 2 ˆ β ∥ 2 is bounded with high probability (boundedness is a consequence of the fact that ∥ Σ 1 / 2 ˆ β ∥ 2 or ∥ Σ 1 / 2 ( ˆ β -β ) ∥ 2 admit finite limits);

(ii) In the component of the ( γ, κ ) plane where the MLE exists, for any null covariate j ∈ [ p ] , the asymptotic normality results (1.2)-(1.3) holds.

Multiclass classification. The goal of this paper is to develop a theory for the asymptotic normality of the multinomial logistic regression MLE (or cross-entropy minimizer) on null covariates when the number of classes, K +1 , is greater than 2 and n, p are of the same order. In other words, we aim to generalize results such as (1.2) or (1.3) for three or more classes. Classification datasets with 3 or more classes are ubiquitous in machine learning (MNIST, CIFAR to name a few), which calls for such multiclass generalizations. In Gaussian mixtures and logistic models, Thrampoulidis et al. [2020] derived characterizations of the performance of of least-squares and class-averaging estimators, excluding cross-entropy minimizers or minimizers of non-linear losses. Loureiro et al. [2021] extended Sur and Candès [2019], Zhao et al. [2022], Salehi et al. [2019] to multiclass classification problems in a Gaussian mixture model, and obtained the fixed-point equations that characterize the performance and empirical distribution of the minimizer of the cross-entropy loss plus a convex regularizer. In the same vein as Loureiro et al. [2021], Cornacchia et al. [2022] studied the limiting

fixed-point equations in a multiclass teacher-student learning model where labels are generated by a noiseless channel with response arg min k ∈{ 1 ,...,K } x T i β k where β k ∈ R p is unknown for each class k . These two aforementioned works assume a multiclass Gaussian mixture model, which is different than the normality assumption for x i used in the present paper. More importantly, these results cannot be readily used for the purpose testing significant covariates (cf. (1.10) below) since solving the fixed-point equations require the knowledge of several unknown parameters, including the limiting spectrum of the mixture covariances and empirical distributions of the mixture means (cf. for instance Corollary 3 in Loureiro et al. [2021]). In the following sections, we fill this gap with a new methodology to test the significance of covariates. This is made possible by developing new asymptotic normality results for cross-entropy minimizers that generalize (1.3), without relying on the low-dimensional fixed-point equations.

Notation. Throughout, I p ∈ R p × p is the identity matrix, for a matrix A ∈ R m × n , A T denotes the transpose of A , A † denotes the Moore-Penrose inverse of A . If A is psd, A 1 / 2 denotes the unique symmetric square root, i.e., the unique positive semi-definite matrix such that ( A 1 / 2 ) 2 = A . The symbol ⊗ denotes the Kronecker product of matrices. Given two matrices A ∈ R n × k , B ∈ R n × q with the same number or rows, ( A,B ) ∈ R n × ( k + q ) is the matrix obtained by stacking the columns of A and B horizontally. If v ∈ R n is a column vector with dimension equal to the number of rows in A , we construct ( A,v ) ∈ R n × ( k +1) similarly. We use 0 n and 1 n to denote the all-zeros vector and all-ones vector in R n , respectively; we do not bold vectors and matrices other than 0 n and 1 n . We may omit the subscript giving the dimension if clear from context; e.g., in I K +1 -11 T K +1 the vector 1 is in R K +1 . The Kronecker product between two matrices is denoted by ⊗ and vec( M ) ∈ R nd is the vectorization operator applied to a matrix M ∈ R n × d . For an integer K ≥ 2 and α ∈ (0 , 1) , the quantile χ 2 K ( α ) is the unique real number satisfying P ( W > χ 2 K ( α )) = α where W has a chi-square distribution with K degrees of freedom. The symbols d - - → and p - - → denote convergence in distribution and in probability.

Throughout, classical asymptotic regime refers to the scenario where the feature dimension p is fixed and the sample size n goes to infinity. In contrast, the term high-dimensional regime refers to the situation where n and p both tend to infinity with the ratio p/n converging to a limit smaller than 1.

1.1 Multinomial logistic regression

Consider a multinomial logistic regression model with K +1 classes. We have n i.i.d. data samples { ( x i , y i ) } n i =1 , where x i ∈ R p is the feature vector and y i = ( y i 1 , ..., y i ( K +1) ) T ∈ R K +1 is the response. Each response y i is the one-hot encoding of a single label, i.e., y i ∈ { 0 , 1 } K +1 with ∑ K +1 k =1 y ik = 1 such that y ik = 1 if and only if the label for i -th observation is k . A commonly used generative model for y i is the multinomial regression model, namely

where B ∗ ∈ R p × ( K +1) is an unknown logistic model parameter and e k ∈ R K +1 , e k ′ ∈ R K +1 are the k -th and k ′ -th canonical basis vectors. The MLE for B ∗ in the model (1.4) is any solution that minimizes the cross-entropy loss,

where L i : R K +1 → R is defined as L i ( u ) = -∑ K +1 k =1 y ik u k +log ∑ K +1 k ′ =1 exp( u k ′ ) . If the solution set in (1.5) is non-empty, we define for each observation i ∈ [ n ] the vector of predicted probabilities ˆ p i = (ˆ p i 1 , ..., ˆ p i ( K +1) ) T with

Our results will utilize the gradient and Hessian of L i evaluated at ˆ B T x i , denoted by

The quantities ( ˆ B , ˆ p i , g i , H i ) can be readily computed from the data { ( x i , y i ) } n i =1 . To be specific, the MLE ˆ B in (1.5) can be obtained by invoking a multinomial regression solver (e.g., sklearn.linear_model.LogisticRegression from Pedregosa et al. [2011]), and the quantities ˆ p i , g i , H i can be further computed from eqs. (1.6) and (1.7) by a few matrix multiplications and application of the softmax function.

Log-odds model and reference class. The matrix B ∗ in (1.4) is not identifiable since the conditional distribution of y i | x i in the model (1.4) remains unchanged if we replace columns of B ∗ by B ∗ -b 1 T K +1 for any b ∈ R p . In order to obtain an identifiable model, a classical and natural remedy is to model the log-odds, here with the class K +1 as the reference class:

where e k is the k -th canonical basis vector of R K , and A ∗ ∈ R p × K is the unknown parameter. The matrix A ∗ ∈ R p × K in log-odds model (1.8) is related to B ∗ ∈ R p × ( K +1) in the model (1.4) by A ∗ = B ∗ ( I K , -1 K ) T . This log-odds model has two benefits: First it is identifiable since the unknown matrix A ∗ is uniquely defined. Second, the matrix A ∗ lends itself well to interpretation as its k -th column represents the contrast coefficient between class k and the reference class K +1 .

The MLE ˆ A of A ∗ in (1.8) is ˆ A = arg min A ∈ R p × K ∑ n i =1 L i (( A, 0 p ) T x i ) . If the solution set in (1.5) is non-empty, ˆ A is related to any solution ˆ B in (1.5) by ˆ A = ˆ B ( I K , -1 K ) T . Equivalently,

for each j ∈ [ p ] and k ∈ [ K ] . If there are three classes (i.e. K +1 = 3 ), this parametrization allows us to draw scatter plots of realizations of √ ne T j ˆ A = ( √ n ˆ A j 1 , √ n ˆ A j, 2 ) as in Figure 1.

1.2 Hypothesis testing for the j -th feature and classical asymptotic normality for MLE

Hypothesis testing for the j -th feature. Our goal is to develop a methodology to test the significance of the j -th feature. Specifically, for a desired confidence level (1 -α ) ∈ (0 , 1) (say, 1 -α = 0 . 95 ) and a given feature j ∈ [ p ] of interest, our goal is to test

Namely, we want to test whether the j -th variable is independent from the response given all other explanatory variables ( x ij ′ , j ′ ∈ [ p ] \ { j } ) . Assuming normally distributed x i and a multinomial model as in (1.4) or (1.8), it is equivalent to test

̸

where e j ∈ R p is the j -th canonical basis vector.

If the MLE ˆ B in (1.5) exists in the sense that the solution set in (1.5) is nonempty, the conjecture that rejecting H 0 when e T j ˆ B is far from 0 K +1 is a reasonable starting point. The important question, then, is to determine a quantitative statement for the informal 'far from 0 K +1 ', similarly to (1.2) or (1.3) in binary logistic regression.

Classical theory with p fixed. If p is fixed and n → ∞ in model (1.8), classical maximum likelihood theory [Van der Vaart, 1998, Chapter 5] provides the asymptotic distribution of the MLE ˆ A , which can be further used to test (1.11). Briefly, if x has the same distribution as any x i , the MLE ˆ A in the multinomial logistic model is asymptotically normal with

is the Fisher information matrix evaluated at the true parameter A ∗ , vec ( · ) is the usual vectorization operator, and π ∗ ∈ R K has random entries π ∗ k = exp( x T A ∗ e k ) / (1 + ∑ K k ′ =1 exp( x T A ∗ e k ′ )) for each k ∈ [ K ] . In particular, under H 0 : e T j A ∗ = 0 T K ,

where S j = ( e T j ⊗ I K ) I -1 ( e j ⊗ I K ) = e T j (cov( x )) -1 e j [ E ( diag ( π ∗ ) -π ∗ π ∗ T )] -1 . When (1.12) holds, by the delta method we also have √ nS -1 / 2 j ˆ A T e j d - - → N ( 0 , I K ) and

where the limiting distribution is chi-square with K degrees of freedom. This further suggests the size α test that rejects H 0 when T j n ( X,Y ) > χ 2 K ( α ) , where T j n ( X,Y ) = n ∥ S -1 / 2 j ˆ A T e j ∥ 2 is the test statistic. If (1.13) holds, this test is guaranteed to have a type I error converging to α . The p-value of this test is given by

where f χ 2 K ( · ) is the density of the chi-square distribution with K degrees of freedom.

As discussed in the introduction, Sur and Candès [2019] showed that in binary logistic regression, classical normality results for the MLE such as (1.12) fail in the high-dimensional regime because the variance in (1.12) underestimates the variability of the MLE even for null covariates; see also the discussion surrounding (1.2). Our goal is to develop, for classification problems with K +1 ≥ 3 classes, a theory that correctly characterize the asymptotic distribution of ˆ A T e j for a null covariate j ∈ [ p ] in the high-dimensional regime.

We present first some motivating simulations that demonstrate the failure of classical normal approximation (1.12) in finite samples. These simulations are conducted for various configurations of ( n, p ) with K + 1 = 3 classes. We fix the true parameter A ∗ and obtain 1000 realizations of ( ˆ A j 1 , ˆ A j 2 ) by independently resampling the data { ( x i , y i ) } n i =1 1000 times. If the result (1.12) holds, then P ( √ n ˆ A T e j ∈ C j α ) → 1 -α , where C j α = { u ∈ R K : ∥ S -1 / 2 j u ∥ ≤ χ 2 K ( α ) } . Figure 1 displays scatter plots of √ n ( ˆ A j 1 , ˆ A j 2 ) along with the boundary of 95% confidence set C j α with α = 0 . 05 . We observe that, across the three different configurations of ( n, p ) , the 95% confidence sets from our theory (Theorem 2.2 presented in next section) cover around 95% of the realizations, while the set C j α from classical theory only covers approximately 30% of the points, which is significantly lower than the desired coverage rate of 95% . Intuitively and by analogy with results in binary classification [Sur and Candès, 2019], this is because the classical theory (1.12) underestimates the variation of the MLE in the high-dimensional regime. Motivated by this failure of classical MLE theory and the results in binary classification [Sur and Candès, 2019, among others], the goal of this paper is to develop a theory for multinomial logistic regression that achieves the following objectives:

20

10

0

10

20

20

10

0

10

20

classical; coverage:30.5%

modern; coverage:94.3%

(a)

(

n, p

) = (2000

,

600)

20

15

10

5

0

5

10

15

20

20

15

10

5

0

5

10

15

20

classical; coverage:35.5%

modern; coverage:95.3%

(b)

(

n, p

) = (3500

,

1000)

20

10

0

10

20

20

10

0

10

20

classical; coverage:28.8%

modern; coverage:95.1%

(c)

(

n, p

) = (5000

,

1500)

Establish asymptotic normality of the multinomial MLE ˆ A T e j for null covariates as n, p → + ∞ simultaneously with a finite limit for n/p .

Develop a valid methodology for hypothesis testing of (1.10) in this regime, i.e., testing for the presence of an effect of a feature j ∈ [ p ] on the multiclass response.

The contribution of this paper is two-fold: (i) For a null covariate j ∈ [ p ] , we establish asymptotic normality results for ˆ A T e j that are valid in the high-dimensional regime where n and p have the same order; (ii) we propose a user-friendly test for assessing the significance of a feature in multiclass classification problems.

2 Main result: asymptotic normality of ˆ B T e j and ˆ A T e j on null covariates

In this section, we present the main theoretical results of our work and discuss their significance. We work under the following assumptions.

Assumption 2.1. For constants δ > 1 , assume that n, p → ∞ with p/n ≤ δ -1 , and that the design matrix X ∈ R n × p has n i.i.d. rows ( x i ) i ∈ [ n ] ∼ N ( 0 , Σ) for some invertible Σ ∈ R p × p . The observations ( x i , y i ) i ∈ [ n ] are i.i.d. and each y i is of the form y i = f ( U i , x T i B ∗ ) for some deterministic function f , deterministic matrix B ∗ ∈ R p × ( K +1) such that B ∗ 1 K +1 = 0 p , and latent random variable U i independent of x i .

Assumption 2.2 (One-hot encoding) . The response matrix Y is in R n × ( K +1) . Its i -th row y i is a one-hot encoded vector, that is, valued in { 0 , 1 } K +1 with ∑ K +1 k =1 y ik = 1 for each i ∈ [ n ] .

The model y i = f ( U i , x T i B ∗ ) for some deterministic f and B ∗ and latent random variable U i in Assumption 2.1 is more general than a specific generative model such as the multinomial logistic conditional probabilities in (1.4), as broad choices for f are allowed. In words, the model y i = f ( U i , x T i B ∗ ) with B ∗ 1 K +1 = 0 p means that y i only depends on x i through a K dimensional projection of x i (the projection on the row-space of B ∗ ). The assumption p/n ≤ δ -1 is more general than assuming a fixed limit for the ratio p/n ; this allows us to cover low-dimensional settings satisfying p/n → 0 as well.

The following assumption requires the labels to be 'balanced': we observe each class at least γn times for some constant γ > 0 . If ( y i ) i ∈ [ n ] are i.i.d. as in Assumption 2.1 with distribution independent of n, p , by the law of large numbers this assumption is equivalent to min k ∈ [ K +1] P ( y ik = 1) > 0 .

Assumption 2.3. There exits a constant γ ∈ (0 , 1 K +1 ] , such that for each k ∈ [ K + 1] , with probability approaching one at least γn observations i ∈ [ n ] are such that y ik = 1 . In other words, P ( ∑ n i =1 I ( y ik = 1) ≥ γn ) → 1 for each k ∈ [ K +1] .

As discussed in item list (i) on page 2, in binary logistic regression, Candès and Sur [2020], Sur and Candès [2019] show that the plane ( p n , ∥ Σ 1 / 2 β ∗ ∥ ) is split by a smooth curve into two connected open components: in one component the MLE does not exist with high probability, while in the other component, with high probability the MLE exists and is bounded in the sense that ∥ Σ 1 / 2 ˆ β ∥ 2 < τ ′ or equivalently 1 n ∥ X ˆ β ∥ 2 < τ for constants τ, τ ′ independent of n, p . The next assumption requires the typical situation of the latter component, in the current multiclass setting: ˆ B in (1.5) exists in the sense that the minimization problem has solutions, and at least one solution is bounded.

Assumption 2.4. Assume P ( ˆ B exists and ∥ X ˆ B ( I K +1 -11 T K +1 ) ∥ 2 F ≤ nτ ) → 1 as n, p → + ∞ for some large enough constant τ .

Note that the validity of Assumption 2.4 can be assessed using the data at hand; if a multinomial regression solver (e.g. sklearn.linear_model.LogisticRegression ) converges 1 and 1 n ∥ X ˆ B ( I K +1 -11 T K +1 ) ∥ 2 F is no larger than a predetermined large constant τ , then we know Assumption 2.4 holds. Otherwise the algorithm does not converge or produces an unbounded estimate: we know Assumption 2.4 fails to hold and we need collect more data.

Our first main result, Theorem 2.1, provides the asymptotic distribution of ˆ B T e j where j ∈ [ p ] is a null covariate, where ˆ B is any minimizer ˆ B of (1.5). Throughout, we denote by Ω the precision matrix defined as Ω = Σ -1 .

Theorem 2.1. Let Assumptions 2.1 to 2.4 be fulfilled. Then for any j ∈ [ p ] such that H 0 in (1.10) holds, and any minimizer ˆ B of (1.5) , we have

where g i = -y i +ˆ p i as in (1.7) and V i = H i -( H i ⊗ x T i )[ ∑ n l =1 H l ⊗ ( x l x T l )] † ( H i ⊗ x i ) .

The proof of Theorem 2.1 is given in Supplementary Section S3. Theorem 2.1 establishes that under H 0 , ˆ B T e j converges to a singular multivariate Gaussian distribution in R K +1 . In (2.1), the two matrices 1 n ∑ n i =1 ( y i -ˆ p i )( y i -ˆ p i ) T and 1 n ∑ n i =1 V i are symmetric with kernel being the linear span of 1 K +1 , and similarly, if a solution exists, we may replace ˆ B by ˆ B ( I K +1 -11 T K +1 ) which is also solution in (1.5). In this case, all matrix-matrix and matrix-vector multiplications, matrix square root and pseudo-inverse in (2.1) happen with row-space and column space contained in the orthogonal component of 1 K +1 , so that the limiting Gaussian distribution in R K +1 is also supported on this K -dimensional subspace.

Since the distribution of the left-hand side of (2.1) is asymptotically pivotal for all null covariates j ∈ [ p ] , Theorem 2.1 opens the door of statistical inference for multinomial logistic regression in high-dimensional settings. By construction, the multinomial logistic estimate ˆ A ∈ R p × K in (1.9) ensures ( ˆ A, 0 p ) is a minimizer of (1.5). Therefore, we can deduce the following theorem from Theorem 2.1.

Theorem 2.2. Define the matrix R = ( I K , 0 K ) T ∈ R ( K +1) × K using block matrix notation. Let Assumptions 2.1 to 2.4 be fulfilled. For ˆ A in (1.9) and any j ∈ [ p ] such that H 0 in (1.10) holds,

where g i is defined in (1.7) and V i is defined in Theorem 2.1. Furthermore, for the same j ∈ [ p ] ,

Note that Equations (2.2) and (2.3) is stated using Ω jj = e T j Σ -1 e j . When Σ is unknown, the quantity Ω jj in above results can be replaced by its consistent estimate ˆ Ω jj defined in (2.5) , and the convergence in distribution results still hold.

Theorem 2.2 is proved in Supplementary Section S4. To the best of our knowledge, Theorem 2.2 is the first result that characterizes the distribution of null MLE coordinate ˆ A T e j in high-dimensional multinomial logistic regression with 3 or more classes. It is worth mentioning that the quantities ( g i , V i , ˆ A ) used in Theorem 2.2 can be readily computed from the data ( X,Y ) . Therefore, Theorem 2.2 lets us test the significance of a specific feature: for testing H 0 , this theorem suggests the test statistic T j n ( X,Y ) in (2.3) and the rejection region E j α def := { ( X,Y ) : T j n ( X,Y ) ≥ χ 2 K ( α ) } . Under the null hypothesis H 0 in (1.10), Theorem 2.2 guarantees P ( ( X,Y ) ∈ E j α ) → α. In other words, the test that rejects H 0 if ( X,Y ) ∈ E j α has type I error converging to α . The p-value of this test is

where f χ 2 K ( · ) is the density of the chi-square distribution with K degrees of freedom.

Unknown Ω jj = e T j Σ -1 e j . If Σ is unknown, we describe a consistent estimate of the quantity Ω jj appearing in (2.1), (2.2), and (2.3). Under the Gaussian Assumption 2.1, the quantity Ω jj is the

0.0

2.5

5.0

7.5

10.0

12.5

0

2

4

6

8

10

12

classical

modern

(a)

(

n, K

) = (4000

,

2)

0

5

10

15

0

2

4

6

8

10

12

14

16

classical

modern

(b)

(

n, K

) = (5000

,

3)

0

5

10

15

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

classical

modern

(c)

(

n, K

) = (6000

,

4)

reciprocal of the conditional variance Var( x ij | x i, -j ) , which is also the noise variance in the linear model of regressing Xe j onto X -j (the submatrix of X excluding the j -th column). According to standard results in linear models, we have Ω jj ∥ [ I n -X -j ( X T -j X -j ) -1 X T -j ] Xe j ∥ 2 ∼ χ 2 n -p +1 . Since χ 2 n -p +1 / ( n -p +1) → 1 almost surely by the strong law of large numbers,

is a consistent estimator of Ω jj . Therefore, the previous asymptotic results in Theorems 2.1 and 2.2 still hold by Slutsky's theorem if we replace Ω jj by the estimate ˆ Ω jj in (2.5).

3 Numerical experiments

This section presents simulations and a real data analysis to examine finite sample properties of the above results and methods. The source code for generating all of the experimental results in this paper can be found in the supplementary material.

Simulation settings. Weset p = 1000 and consider different combinations of ( n, K ) . The covariance matrix Σ is specified to be the correlation matrix of an AR(1) model with parameter ρ = 0 . 5 , that is, Σ = (0 . 5 | i -j | ) p × p . We generate the regression coefficients A ∗ ∈ R p × K once and for all as follows: sample A 0 ∈ R p × K with first ⌈ p/ 4 ⌉ rows being i.i.d. N ( 0 , I K ) , and set the remaining rows to 0. We then scale the coefficients by defining A ∗ = A 0 ( A T 0 Σ A 0 ) -1 / 2 so that A ∗ T Σ A ∗ = I K . With this construction, the p -th variable is always a null covariate , and we use this null coordinate j = p to demonstrate the effectiveness of our theoretical results presented in Theorem 2.2 and the suggested test for testing H 0 as described in (1.10). Using the above settings, we generate the design matrix X ∈ R n × p from N (0 , Σ) , and then simulate the labels from a multinomial logistic model as given in (1.8), using the coefficients A ∗ ∈ R p × K . For each simulation setting, we perform 5,000 repetitions.

Assessment of χ 2 approximations. To assess the χ 2 approximation (2.3) from this paper and that of the classical theory (1.13), we compute the two χ 2 K test statistics for each sample ( x i , y i ) n i =1 . Figure 2 shows the empirical quantiles of the two statistics versus the χ 2 K distribution quantiles. The results demonstrate that the quantiles (in blue) from our high-dimensional theory closely match the 45-degree line (in red), whereas the quantiles (in orange) from the classical theory significantly deviate from the 45-degree line. These findings highlight the accuracy of our proposed χ 2 approximation (2.3) over the classical result (1.13) when p is not sufficiently small compared to n .

Uniformity of null p-values. Recall that the p-value from the classical test (1.13) is given by (1.14), while the p-value from this paper taking into account high-dimensionality is given by (2.4). Figure 3 displays the histograms of these two sets of p-values out of 5000 repetitions. The results in Figure 3 show that the p-values obtained from the classical test deviate significantly from the uniform distribution, with a severe inflation in the lower tail. This indicates that the classical test tends to produce large type I errors due to the excess of p-values close to 0. In contrast, the p-values proposed in this paper exhibit a uniform distribution, further confirming the effectiveness and applicability of the theory in Theorem 2.2 for controlling type I error when testing for null covariates with (1.10).

0.0

0.2

0.4

0.6

0.8

1.0

p-values

0

500

1000

1500

2000

2500

frequency

classical

modern

(a)

(

n, K

) = (4000

,

2)

0.0

0.2

0.4

0.6

0.8

1.0

p-values

0

500

1000

1500

2000

2500

3000

frequency

classical

modern

(b)

(

n, K

) = (5000

,

3)

0.0

0.2

0.4

0.6

0.8

1.0

p-values

0

500

1000

1500

2000

2500

3000

frequency

classical

modern

(c)

(

n, K

) = (6000

,

4)

Unknown Ω jj . In the situation where the covariance matrix Σ is unknown, we can estimate the diagonal element Ω jj = e T j Σ -1 e j by ˆ Ω jj defined in (2.5). To evaluate the accuracy of the normal and chi-square approximations and the associated test with Ω jj replaced by ˆ Ω jj , we conduct simulations similar to those in Figures 2 and 3, but we replace Ω jj with its estimate ˆ Ω jj . The results are presented in Figure S1. The plots are visually indistinguishable from the plots using Ω jj . These confirm that the chi-square approximation and the associated test using ˆ Ω jj are accurate.

Non-Gaussian covariates and unknown Ω jj . Although our theory assumes Gaussian covariates, we expect that the same results hold for other distributions with sufficiently light tails. To illustrate this point, we consider the following two types of non-Gaussian covariates: (i) The design matrix X has i.i.d. Rademacher entries, i.e., P ( x ij = ± 1) = 1 2 , (ii) Each x ij takes on values 0, 1 and 2 with respectively probabilities a 2 j , 2 a j (1 -a j ) , and (1 -a j ) 2 , where a j varies in [0 . 25 , 0 . 75] . Each columns of X are then centered and normalized to have 0 mean and unit variance. This generation of non-Gaussian covariates is adopted from single-nucleotide poly-morphisms (SNPs) example in Sur and Candès [2019]. For these two types of non-Gaussian covariates, we further rescale the feature vectors to ensure that x i has the same covariance as in the Gaussian case at the beginning of Section 3, that is Σ = (0 . 5 | i -j | ) p × p . We present the Q-Q plots in Figure S2 using the same settings as in Figure S1, with the only difference being that the covariates in Figure S2 are non-Gaussian distributed. The Q-Q plots of T j n ( X,Y ) in (2.3) plotted in Figure S2 still closely match the diagonal line. These empirical successes suggest that the normal and χ 2 K approximations (2.1)-(2.3) apply to a wider range of covariate distributions beyond normally distributed data.

Real data example. We conduct a real data analysis by applying the proposed test to heart disease data from the UCI Machine Learning Repository (link: http://archive.ics.uci.edu/ml/machine-learningdatabases/heart-disease/processed.cleveland.data). After standard data-cleaning processes, the dataset has 297 instances with 13 features, including age, sex, and other attributes. The response variable was transformed into 3 classes (0, 1, and 2) after converting the labels 3 and 4 to 2. To demonstrate the validity of the proposed significance test, we generate a noise variable from a standard normal distribution, resulting in a dataset with 297 instances and 14 variables. We test the significance of this noise variable using both the proposed chi-square test and the classical test, repeating the experiment 10,000 times. The type I error of our proposed test is 0.0508, aligning well with the desired type I error of 0.05. In contrast, the classical test exhibits a type I error of 0.0734, significantly exceeding the desired rate of 0.05. These results confirm the validity of our proposed test on real data corrupted with fake covariate, while the classical test concludes that the fake covariate is significant in more than 7% of experiments, leading to false discoveries exceeding the desired 0.05 type I error.

4 Discussion and future work

Multinomial logistic regression estimates and their p-values are ubiquitous throughout the sciences for analyzing the significance of explanatory variables on multiclass responses. Following the seminal work of Sur and Candès [2019] in binary logistic regression, this paper develops the first valid tests and p-values for multinomial logistic estimates when p and n are of the same order. For 3 or more

classes, this methodology and the corresponding asymptotic normality results in Theorems 2.1 and 2.2 are novel and provide new understanding of multinomial logistic estimates (also known as crossentropy minimizers) in high-dimensions. We expect similar asymptotic normality and chi-square results to be within reach for loss functions different than the cross-entropy or a different model for the response y i ; for instance Section S1 provides an extension to the q -repeated measurements model, where q responses are observed for each feature vector x i .

Let us point a few follow-up research directions that we leave open for future work. A first open problem regards extensions of our methodology to confidence sets for e T j B ∗ when H 0 in (1.10) is violated for the j -th covariate. This would require more stringent assumptions on the generative model than Assumption 2.1 as B ∗ there is not identifiable (e.g., modification of both B ∗ and f ( · , · ) in Assumption 2.1 is possible without changing y i ). A second open problem is to relate this paper's theory to the fixed-point equations and limiting Gaussian model obtained in multiclass models, e.g., Loureiro et al. [2021]. While it may be straightforward to obtain the limit of 1 n ∑ n i =1 g i g T i and of the empirical distribution of the rows of ˆ A in this context (e.g., using Corollary 3 in Loureiro et al. [2021]), the relationship between the fixed-point equations and the matrix 1 n ∑ n i =1 V i appearing in (2.1) is unclear and not explained by typical results from this literature. A third open problem is to characterize the exact phase transition below which the multinomial logistic MLE exists and is bounded with high-probability (Assumption 2.4); while this is settled for two classes [Candès and Sur, 2020] and preliminary results are available for 3 or more classes [Loureiro et al., 2021, Kini and Thrampoulidis, 2021], a complete understanding of this phase transition is currently lacking. A last interesting open problem is to prove that our theory extend to non-Gaussian data, as observed in simulations. This challenging problem is often referred to as 'universality' and has received intense attention recently [Gerace et al., 2022, Han and Shen, 2022, Montanari and Saeed, 2022, Pesce et al., 2023, Dandi et al., 2023], showing that in several settings of interest (although none exactly matching the one considered in the present paper), the asymptotic behavior of the minimizers is unchanged if the distribution of the covariates is modified from normal to another distribution with the same covariance.

Acknowledgments and Disclosure of Funding

P.C. Bellec's research was partially supported by the NSF Grant DMS-1945428. The authors acknowledge the Office of Advanced Research Computing (OARC) at Rutgers, The State University of New Jersey for providing access to the Amarel cluster and associated research computing resources that have contributed to the results reported here. URL: https://oarc.rutgers.edu.

References

Pierre C Bellec. Observable adjustments in single-index models for regularized m-estimators. arXiv preprint arXiv:2204.06990 , 2022.

Pierre C. Bellec and Cun-Hui Zhang. Second-order Stein: SURE for SURE and other applications in high-dimensional inference. Ann. Statist. , 49(4):1864-1903, 2021. ISSN 0090-5364.

Raphael Berthier, Andrea Montanari, and Phan-Minh Nguyen. State evolution for approximate message passing with non-separable functions. Information and Inference: A Journal of the IMA , 9(1):33-79, 2020.

Emmanuel J. Candès and Pragya Sur. The phase transition for the existence of the maximum likelihood estimate in high-dimensional logistic regression. The Annals of Statistics , 48(1):27 - 42, 2020. .

Elisabetta Cornacchia, Francesca Mignacco, Rodrigo Veiga, Cédric Gerbelot, Bruno Loureiro, and Lenka Zdeborová. Learning curves for the multi-class teacher-student perceptron. arXiv preprint arXiv:2203.12094 , 2022.

Jan Salomon Cramer. The origins of logistic regression. Tinbergen Institute Working Paper , 2002.

Yatin Dandi, Ludovic Stephan, Florent Krzakala, Bruno Loureiro, and Lenka Zdeborová. Universality laws for gaussian mixtures in generalized linear models. arXiv preprint arXiv:2302.08933 , 2023.

Kenneth R Davidson and Stanislaw J Szarek. Local operator theory, random matrices and banach spaces. Handbook of the geometry of Banach spaces , 1(317-366):131, 2001.

Oliver Y Feng, Ramji Venkataramanan, Cynthia Rush, Richard J Samworth, et al. A unifying tutorial on approximate message passing. Foundations and Trends® in Machine Learning , 15(4):335-536, 2022.

Federica Gerace, Florent Krzakala, Bruno Loureiro, Ludovic Stephan, and Lenka Zdeborová. Gaussian universality of linear classifiers with random labels in high-dimension. arXiv preprint arXiv:2205.13303 , 2022.

Cédric Gerbelot and Raphaël Berthier. Graph-based approximate message passing iterations. arXiv preprint arXiv:2109.11905 , 2021.

Qiyang Han and Yandi Shen. Universality of regularized regression estimators in high dimensions. arXiv preprint arXiv:2206.07936 , 2022.

Ganesh Ramachandra Kini and Christos Thrampoulidis. Phase transitions for one-vs-one and one-vsall linear separability in multiclass gaussian mixtures. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 4020-4024. IEEE, 2021.

Bruno Loureiro, Gabriele Sicuro, Cédric Gerbelot, Alessandro Pacco, Florent Krzakala, and Lenka Zdeborová. Learning gaussian mixtures with generalized linear models: Precise asymptotics in high-dimensions. Advances in Neural Information Processing Systems , 34:10144-10157, 2021.

Xiaoyi Mai, Zhenyu Liao, and Romain Couillet. A large scale analysis of logistic regression: Asymptotic performance and new insights. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 3357-3361. IEEE, 2019.

Andrea Montanari and Basil N Saeed. Universality of empirical risk minimization. In Conference on Learning Theory , pages 4310-4312. PMLR, 2022.

Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research , 12:2825-2830, 2011.

Luca Pesce, Florent Krzakala, Bruno Loureiro, and Ludovic Stephan. Are gaussian data all you need? extents and limits of universality in high-dimensional generalized linear estimation. arXiv preprint arXiv:2302.08923 , 2023.

Fariborz Salehi, Ehsan Abbasi, and Babak Hassibi. The impact of regularization on high-dimensional logistic regression. Advances in Neural Information Processing Systems , 32, 2019.

Pragya Sur and Emmanuel J. Candès. A modern maximum-likelihood theory for high-dimensional logistic regression. Proc. Natl. Acad. Sci. USA , 116(29):14516-14525, 2019. ISSN 0027-8424. .

Kai Tan, Gabriel Romon, and Pierre C Bellec. Noise covariance estimation in multi-task highdimensional linear models. arXiv preprint arXiv:2206.07256 , 2022.

Christos Thrampoulidis, Samet Oymak, and Mahdi Soltanolkotabi. Theoretical insights into multiclass classification: A high-dimensional asymptotic view. Advances in Neural Information Processing Systems , 33:8907-8920, 2020.

Aad W Van der Vaart. Asymptotic statistics , volume 3. Cambridge university press, 1998.

J Leo van Hemmen and Tsuneya Ando. An inequality for trace ideals. Communications in Mathematical Physics , 76:143-148, 1980.

Steve Yadlowsky, Taedong Yun, Cory Y McLean, and Alexander D'Amour. Sloe: A faster method for statistical inference in high-dimensional logistic regression. Advances in Neural Information Processing Systems , 34:29517-29528, 2021.

Qian Zhao, Pragya Sur, and Emmanuel J Candes. The asymptotic distribution of the mle in highdimensional logistic models: Arbitrary covariance. Bernoulli , 28(3):1835-1861, 2022.

Ji Zhu and Trevor Hastie. Classification of gene microarrays by penalized logistic regression. Biostatistics , 5(3):427-443, 2004.

Supplementary Material of 'Multinomial Logistic Regression: Asymptotic Normality on Null Covariates in High-Dimensions'

Let us define some standard notation that will be used in the rest of this supplement. For a vector v ∈ R n , let ∥ v ∥ ∞ = max i ∈ [ n ] | v i | denote the infinity norm of vector v . If A is symmetric, we define λ min ( A ) and ∥ A ∥ op as the minimal and maximal eigenvalues of A , respectively. For two symmetric matrices A,B of the same size, we write A ⪯ B if and only if B -A is positive semi-definite.

Diagram: Organization of the proofs

The following diagram summarizes the different theorems and lemmas, and the relationships between them.

̸

Theorem 2.2

Asymptotic normality for

ˆ

A

T

e

j

on null covariates,

where

ˆ

A

∈

R

p

×

K

is the

multinomial logistic MLE

with

class

K

+ 1

fixed

as the reference class (see

(1.9)).

Theorem 2.1

Asymptotic normality for

ˆ

B

T

e

j

on null covariates,

where

ˆ

B

∈

R

p

×

(

K

+1)

is

the

multinomial

logistic

MLE in (1.5).

Theorem S3.1

Asymptotic normality for

ˆ

B

T

e

j

on null covariates,

where

ˆ

B

∈

R

p

×

K

is the

multinomial logistic MLE

using the parameter space

from Section S3.1.

The

proof

uses

that

the

conditions

in

The-

orem

S5.1

on

the

loss

function are satisfied by

the cross-entropy.

K

-dimensional

orthogo-

nal

parametrization

de-

fined by the matrix the

Q

Section S3.1 defines the

matrix

Q

∈

R

(

K

+1)

×

K

and

discusses

a

conve-

nient

parametrization

of

the

model

isometric

to

the subspace orthogonal

to

1

K

+1

.

Control of

g

i

and

H

i

for

the cross-entropy loss

Lemmas S6.1 and S6.2

give

deterministic

ar-

guments

to

control

the

gradients

and

Hessians

of

the

cross-entropy

loss.

Lemma S6.4 con-

trols

the

Hessian of the

cross-entropy loss at the

minimizer,

in

a specific

high-probability

event.

Lemma S6.3 defines this

high-probability event.

Asymptotic normality on

null covariates for general

loss functions,

Σ

=

I

p

.

Deduced

from

Theo-

rem

S5.2

by

rotational

invariance.

Theorem S5.2

Asymptotic normality on

null covariates for general

loss functions,

Σ =

I

p

.

Normal and

χ

2

approxi-

mations for random vari-

ables defined as a differen-

tiable function of standard

normal vectors.

Lemma

computes

the derivatives of the min-

imizer with respect to

X

,

used in the proof of Theo-

S1 Extension: q repeated measurements

Let integer q ≥ 1 be a constant independent of n, p . Our results readily extend if q labels are observed for each observed feature vector x i , and the corresponding q one-hot encoded vectors are averaged into y i ∈ { 0 , 1 q , 2 q , ..., 1 } K +1 . Concretely, for each observation i ∈ [ n ] , q i.i.d. labels ( Y i m ) m ∈ [ q ] are observed with each Y i m ∈ { 0 , 1 } K +1 one-hot encoded and y ik = 1 q ∑ q m =1 Y m ik , for instance in a repeated multinomial regression model with P ( Y m ik = 1 | x i ) equal to right-hand side of (1.4). In this case where ( Y i m ) m ∈ [ q ] are i.i.d., Assumption 2.3 is satisfied by the law of large numbers if min k ∈ [ K +1] P ( Y m ik = 1) > 0 since q is constant. For this q repeated measurements model, the negative log-likelihood function of a parameter B ∈ R p × ( K +1) is

where the first equality uses y ik = 1 q ∑ q m =1 Y m ik , the second equality uses ∑ K +1 k =1 y ik = 1 under the following Assumption S1.1, and the last equality uses the definition of L i after (1.5).

Assumption S1.1. For all i ∈ [ n ] , the response y i is in { 0 , 1 /q, 2 /q, ..., 1 } K +1 with ∑ K +1 k =1 y ik = 1 .

In such repeated measurements model, we replace Assumption 2.2 with Assumption S1.1 under which the following Theorem S1.1 holds.

Theorem S1.1. Let q ≥ 2 be constant. Let Assumptions S1.1, 2.1, 2.3 and 2.4 be fulfilled. For any j ∈ [ p ] such that H 0 in (1.10) holds, we have the convergence in distribution (2.1) , (2.2) and (2.3) .

Proof of Theorem S1.1. Under the assumptions in Theorem S1.1, the MLE ˆ B for this q repeated measurements model is the minimizer of the optimization problem

as in (1.5). Similar to the non-repeated model, the MLE ˆ A for the identifiable log-odds model can be expressed as

The only difference between this q repeated measurements model and the non-repeated model considered in the main text is that the response y ik for this q repeated measurements model is now valued in { 0 , 1 /q, 2 /q, ..., 1 } . Because the proofs of Theorems 2.1 and 2.2 do not require the value of y ik to be { 0 , 1 } -valued. Theorem S1.1 can be proved by the same arguments used in the proof of Theorems 2.1 and 2.2.

S2 Implementation details and additional figures

The pivotal quantities in our main results Theorems 2.1 and 2.2 involve only observable quantities that can be computed from the data ( x i , y i ) i ∈ [ n ] . In this section we provide an efficient way of computing the matrix V i appearing in Theorems 2.1 and 2.2.

Fast computation of V i . Recall the definition of V i in Theorem 2.1,

The majority of computational cost in calculating V i lies in the step of calculating its second term

Here we provide an efficient way to compute this term using the Woodbury matrix identity. Since H i 1 K +1 = 0 K +1 , we have ker( H i ⊗ ( x i x T i )) is the span of { 1 K +1 ⊗ e j : j ∈ [ p ] } , where 1 K +1 is the all-ones vector in R K +1 . Therefore, the second term in V i can be rewritten as

We now apply the Woodbury matrix identity to compute the matrix inversion in the above display. Recall H i = diag(ˆ p i ) -ˆ p i ˆ p T i , we have

Let A = ∑ K +1 k =1 ( e k e T k ) ⊗ ( ∑ n i =1 ˆ p ik x i x T i ) , and U ∈ R p ( K +1) × ( n + p ) with the first n columns being (ˆ p i ⊗ x i ) i ∈ [ n ] and the following p columns ( 1 ⊗ e j ) j ∈ [ p ] . Then the term we want to invert is A -UU T , where A is a block diagonal matrix and can be inverted by inverting each block separately. By the Woodbury matrix identity, we have

The gain of using the above formula is significant for large K : instead of inverting the p ( K +1) × p ( K +1) matrix ∑ n l =1 H l ⊗ ( x l x T l ) in the left-hand side, the right-hand side only requires to invert a block diagonal matrix A and a ( n + p ) × ( n + p ) matrix -I n + p + U T A -1 U .

0.0

2.5

5.0

7.5

10.0

12.5

0

2

4

6

8

10

12

classical

modern

0

5

10

15

0

2

4

6

8

10

12

14

16

classical

modern

0

5

10

15

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

classical

modern

0.0

0.2

0.4

0.6

0.8

1.0

p-values

0

500

1000

1500

2000

2500

frequency

classical

modern

(a)

(

n, K

) = (4000

,

2)

0.0

0.2

0.4

0.6

0.8

1.0

p-values

0

500

1000

1500

2000

2500

3000

frequency

classical

modern

(b)

(

n, K

) = (5000

,

3)

0.0

0.2

0.4

0.6

0.8

1.0

p-values

0

500

1000

1500

2000

2500

3000

frequency

classical

modern

(c)

(

n, K

) = (6000

,

4)

0.0

2.5

5.0

7.5

10.0

12.5

0

2

4

6

8

10

12

classical

modern

0

5

10

15

0

2

4

6

8

10

12

14

16

classical

modern

0

5

10

15

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

classical

modern

0.0

2.5

5.0

7.5

10.0

12.5

0

2

4

6

8

10

12

classical

modern

(a)

(

n, K

) = (4000

,

2)

0

5

10

15

0

2

4

6

8

10

12

14

16

classical

modern

(b)

(

n, K

) = (5000

,

3)

0

5

10

15

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

classical

modern

(c)

(

n, K

) = (6000

,

4)

20

10

0

10

20

20

10

0

10

20

classical; coverage:30.5%

modern; coverage:94.3%

(a)

q

= 1

10

5

0

5

10

10

5

0

5

10

classical; coverage:66.7%

modern; coverage:94.7%

(b)

q

= 2

8

6

4

2

0

2

4

6

8

8

6

4

2

0

2

4

6

8

classical; coverage:76.5%

modern; coverage:94.9%

(c)

q

= 4

S3 Proof of Theorem 2.1

Before proving Theorem 2.1, we present another parametrization of the multinomial logistic regression model. The asymptotic theory of MLE for this new parametrized multinomial logistic model will be used to prove Theorem 2.1.

S3.1 Another parametrization of multinomial logistic regression

Recall the symbol ' ∈ " is used in (1.5) to emphasize that the minimizer ˆ B in (1.5) is not unique: if ˆ B is a minimizer of (1.5) then ˆ B -b 1 T K +1 is also a minimizer of (1.5), for any b ∈ R p and the all-ones vector 1 K +1 in R K +1 . Besides the log-odds model (1.8), here we consider another identifiable parametrization of multinomial logistic regression, whose unknown parameter, denoted by B ∗ , is in R p × K .

Orthogonal complement. To obtain an identifiable multinomial logistic regression model from (1.4), we consider the symmetric constraint B ∗ 1 = ∑ K +1 k =1 B ∗ e k = 0 as in [Zhu and Hastie, 2004], thus B ∗ = B ∗ ( I K +1 -11 T K +1 ) , where 1 is the all-ones vector in R K +1 . Let Q ∈ R ( K +1) × K be any matrix such that

We fix one choice of Q satisfying (S3.1) throughout this supplement. Let B ∗ = B ∗ Q , then B ∗ = B ∗ Q T and the model (1.4) can be parameterized using B ∗ as

The multinomial logistic MLE of B ∗ in (S3.2) is given by

where L i : R K → R is defined by L i ( u ) = L i ( Qu ) for all u ∈ R K . By this construction, we have ˆ B = ˆ B Q for any minimizer ˆ B of (1.5). Furthermore, by the chain rule using the expressions (1.7), the gradient and Hessian of L i evaluated at ˆ B T x i are

Throughout, we use serif upright letters to denote quantities defined on the unidentifiable parameter space R p × ( K +1) :

and the normal italic font to denote analogous quantities for the identifiable parameter space R p × K :

Theorem S3.1 provides the asymptotic normality and the chi-square approximation of null MLE coordinates in high-dimensions where n, p →∞ with the ratio n/p converging to a finite limit.

Theorem S3.1 (Proof is given on page 27) . Let Assumptions 2.1, 2.3 and 2.4 be fulfilled. Assume that either Assumption 2.2 or Assumption S1.1 holds. Then for any j ∈ [ p ] such that H 0 in (1.10) holds,

where V i = H i -( H i ⊗ x T i )[ ∑ n l =1 H l ⊗ ( x l x T l )] -1 ( H i ⊗ x i ) .

A direct consequence of (S3.5) is the χ 2 result,

The proof of Theorem S3.1 is deferred to Section S5 and Section S6. In the next subsection, we prove Theorem 2.1 using Theorem S3.1.

S3.2 Proof of Theorem 2.1

We restate Theorem 2.1 for convenience.

Theorem 2.1. Let Assumptions 2.1 to 2.4 be fulfilled. Then for any j ∈ [ p ] such that H 0 in (1.10) holds, and any minimizer ˆ B of (1.5) , we have

where

g

i

=

-

y

i

+ˆ

p

i

as in

(1.7)

and

V

i

=

H

i

-

(

H

i

⊗

x

T

i

)[

∑

n

l

=1

H

l

⊗

(

x

l

x

T

l

)]

(

H

i

⊗

x

i

)

.

The proof of Theorem 2.1 is a consequence of Theorem S3.1. To begin with, we state the following useful lemma.

Lemma S3.2. For V i and V i defined in Theorems 2.1 and S3.1, we have V i = Q T V i Q .

where the penultimate equality is proved as follows.

Let A = Q ⊗ I p and D = ∑ n i =1 H i ⊗ ( x i x T i ) only in the remaining of this proof. It remains to prove A [ A T D A ] -1 A T = D † . (S3.7)

Since H i 1 = 0 , we have D ( 1 ⊗ I p ) = 0 . Since Q T 1 = 0 by definition of Q , we have A T ( 1 ⊗ I p ) = 0 . If we write the eigen-decomposition of D as D = ∑ pK i =1 λ i u i u T i , then u T i ( 1 ⊗ I p ) = 0 . Hence, with v i = A T u i ,

†

Since v T i v i ′ = u T i AA T u i ′ = u T i [( I K +1 -11 T K +1 ) ⊗ I p ] u i ′ = u T i u i ′ = I ( i = i ′ ) , we have

where the second equality uses Av i = AA T u i = u i . The proof of (S3.7) is complete.

Now we are ready to prove that Theorem 2.1 is a consequence of Theorem S3.1.

Proof of Theorem 2.1. By definition of g i and V i , we have 1 T g i = 0 and 1 T V i = 0 T . Thus, we have QQ T g i = g i and QQ T V i = V i . Therefore, we can rewrite the left-hand side of (2.1) (without √ n Ω -1 / 2 jj ) as

where the first equality uses QQ T g i = g i and QQ T V i = V i , the second equality uses g i = Q T g i and V i = Q T V i Q from Lemma S3.2, the third equality follows from the same argument of (S3.7), and the last equality uses Q T Q = I K and ˆ B = ˆ B Q .

Therefore, Theorem S3.1 implies that the limiting covariance for the left-hand side of (2.1) is QQ T = I K -11 T K +1 . This completes the proof.

S4 Proof of Theorem 2.2

We restate Theorem 2.2 for convenience.

Theorem 2.2. Define the matrix R = ( I K , 0 K ) T ∈ R ( K +1) × K using block matrix notation. Let Assumptions 2.1 to 2.4 be fulfilled. For ˆ A in (1.9) and any j ∈ [ p ] such that H 0 in (1.10) holds,

where g i is defined in (1.7) and V i is defined in Theorem 2.1. Furthermore, for the same j ∈ [ p ] ,

Note that Equations (2.2) and (2.3) is stated using Ω jj = e T j Σ -1 e j . When Σ is unknown, the quantity Ω jj in above results can be replaced by its consistent estimate ˆ Ω jj defined in (2.5) , and the convergence in distribution results still hold.

The proof is a direct consequence of Theorem 2.1.

Proof of Theorem 2.2. By definition of ˆ A in (1.9), we have ˆ A = ˆ B ( I K , -1 K ) T and

which is of the form ˆ B -b 1 T with b = ˆ B e K +1 . Therefore, ˆ A ( I K , 0 K ) is also a solution of (1.5). Taking ˆ B in Theorem 2.1 to be ˆ A ( I K , 0 K ) = ˆ AR T gives the desired χ 2 result (2.3) and

Multiplying ( R T ( I K +1 -11 T K +1 ) R ) -1 / 2 R T to the left of the above display gives the desired normality result (2.2) by observing ( R T ( I K +1 -11 T K +1 ) R ) -1 / 2 = ( I K + 1 K 1 T K √ K +1+1 ) . This completes the proof.

S5 Preliminary results for proving Theorem S3.1

S5.1 Results for general loss functions

In this subsection, we will work under the following assumptions with a general convex loss function. Later in Section S6, we will apply the general results of this subsection to the multinomial logistic loss discussed in Section S3.1.

Assumption S5.1. Suppose we have data ( Y, X ) , where Y ∈ R n × ( K +1) with rows ( y 1 , ..., y n ) , and X ∈ R n × p has i.i.d. rows ( x 1 , ..., x n ) with x i ∼ N ( 0 , Σ) and invertible Σ . The observations ( y i , x i ) i ∈ [ n ] are i.i.d. and y i has the form y i = f ( U i , x T i B ∗ ) for some deterministic function f , deterministic B ∗ ∈ R p × K , and latent random variable U i independent of x i . Assume p/n ≤ δ -1 < 1 .

Assumption S5.2. Given data ( Y, X ) , consider twice continuously differentiable and strictly convex loss functions ( L i ) i ∈ [ n ] with each L i : R K → R depending on y i but not on x i .

Provided that the following minimization problem admits a solution, define

Define for each i ∈ [ n ] ,

so that g i ( Y, X ) ∈ R K and H i ( Y, X ) ∈ R K × K . Define

so that G ( Y, X ) ∈ R n × K and V ( Y, X ) ∈ R K × K . If the dependence on data ( Y, X ) is clear from context, we will simply write ˆ B , g i , H i G , and V .

Theorem S5.1. Let Assumptions S5.1 and S5.2 be fulfilled. Let c ∗ , m ∗ , m ∗ , K be positive constants independent of n, p . Let U ∗ ⊂ R p × ( K +1) × R n × p be an open set satisfying

If { ( Y, X ) ∈ U ∗ } , then the minimizer ˆ B ( Y, X ) in Assumption S5.2 exists, H i ⪯ I K for each i ∈ [ n ] , 1 n ∑ n i =1 H i ( Y, X ) ⊗ ( x i x T i ) ⪰ c ∗ ( I K ⊗ Σ) and m ∗ I K ⪯ 1 n G ( Y, X ) T G ( Y, X ) ⪯ m ∗ I K .

For any { ( Y, X ) , ( Y, ˜ X ) } ⊂ U ∗ , ∥ G ( Y, X ) -G ( Y, ˜ X ) ∥ F ≤ L ∥ ( X -˜ X )Σ -1 / 2 ∥ F holds for some positive constant L .

Then for any j ∈ [ p ] such that e T j B ∗ = 0 T K , there exists a random variable ξ ∈ R K such that

and P ( ∥ ξ ∥ 2 > χ 2 K ( α )) ≤ α for all α ∈ (0 , 1) , C is a positive constant depending on ( c ∗ , m ∗ , m ∗ , K, L ) only. If additionally P (( Y, X ) ∈ U ∗ ) → 1 , then ξ in the previous display satisfies ξ d - - → N ( 0 , I K ) and

The proof of Theorem S5.1 is given in next subsection.

S5.2 Proof of Theorem S5.1

In this subsection and next subsection, we will slightly abuse the notations A ∗ and ˆ A , which have different definitions than the definitions in the main text.

Let Σ 1 / 2 B ∗ = ∑ K k =1 s k u k v T k be the singular value decomposition of Σ 1 / 2 B ∗ , where u 1 , ..., u K are the left singular vectors and v 1 , ..., v k the right singular vectors. If Σ 1 / 2 B ∗ is of rank strictly less than K , we allow some s k to be equal to 0 so that Σ 1 / 2 B ∗ = ∑ K k =1 s k u k v T k still holds with orthonormal ( u 1 , ..., u K ) and orthonormal ( v 1 , ..., v K ) . We consider an orthogonal matrix ˜ P ∈ R p × p such that

Since e T j B ∗ = 0 T implies e T j Σ -1 / 2 u k = 0 , we can always find a matrix ˜ P satisfying (S5.1). From now on we fix this matrix ˜ P and consider the following change of variable,

It immediately follows that Z has i.i.d. N (0 , 1) entries and the first p -K rows of A ∗ are all zeros. Since the response y i has the expression y i = f ( U i , x T i B ∗ ) , Y is unchanged by the change of variable (S5.2) from ZA ∗ = XB ∗ . We now work on the multinomial logistic estimation with data ( Y, Z ) and the underlying coefficient matrix A ∗ in (S5.2). Parallel to the estimate ˆ B of B ∗ in Assumption S5.2, we define the estimate of A ∗ using data ( Y, Z ) as

where z i = Z T e i is the i -th row of Z . By construction, we have ˆ A = ˜ P Σ 1 / 2 ˆ B , hence Z ˆ A = X ˆ B and e T 1 ˆ A = e T j ˆ B/ √ Ω jj . Furthermore, the quantities depending on ( Y, X ˆ B ) remain unchanged after the change of variable. In particular, the gradient and Hessian

are unchanged. It follows that the matrix G and V are unchanged. Therefore, we have

In conclusion, with the change of variables (S5.2), we only need to prove Theorem S5.1 in the special case, where the design matrix X i.i.d. N (0 , 1) entries and the response Y is independent of the first p -K columns of X . To this end, we introduce the following Theorem S5.2, and the proof of Theorem S5.1 is a consequence of Theorem S5.2 as it proves the desired result for e T 1 ˆ AV ( G T G ) -1 / 2 .

Theorem S5.2. Let c ∗ , m ∗ , m ∗ , K be constants independent of n, p . Let Z ∈ R n × p have i.i.d. rows ( z 1 , ..., z n ) with z i ∼ N ( 0 , I p ) . Let y 1 , ..., y n ∈ R ( K +1) such that ( y 1 , ..., y n ) is independent of the first p -K columns of Z . Consider twice continuously differentiable and strictly convex loss

functions ( L i ) i =1 ,...,n with each L i : R K → R depending on y i but not on z i and define, provided that the minimizer admits a solution,

G ( Y, Z ) = ∑ n i =1 e i g i ( Y, Z ) T ∈ R n × K , and V ( Y, Z ) = ∑ n i =1 ( H i -( H i ⊗ z T i ) [∑ n l =1 H l ⊗ ( z l z T l ) ] † ( H i ⊗ z i ) ) ∈ R K × K , where we dropped the dependence of H i on ( Y, Z ) for simplicity.

Let O ⊂ R n × ( K +1) × R n × p be an open set satisfying

If ( Y, Z ) ∈ O , then the minimizer ˆ A ( Y, Z ) exists, H i ⪯ I K for each i ∈ [ n ] , c ∗ I pK ⪯ 1 n ∑ n i =1 H i ( Y, Z ) ⊗ ( z i z T i ) , and m ∗ I K ⪯ 1 n ∑ n i =1 G ( Y, Z ) T G ( Y, Z ) ⪯ m ∗ I K .

With the notation G ( Y, Z ) = ∑ n i =1 e i g i ( Y, Z ) T , we have if two Z, ˜ Z ∈ R n × p satisfy { ( Y, Z ) , ( Y, ˜ Z ) } ⊂ O then ∥ G ( Y, Z ) -G ( Y, ˜ Z ) ∥ ≤ L ∥ Z -˜ Z ∥ .

For e 1 ∈ R p the first canonical basis vector, there exists a random variable ξ ∈ R K such that

and P ( ∥ ξ ∥ 2 > χ 2 K ( α )) ≤ α for all α ∈ (0 , 1) , C is a positive constant depending on ( c ∗ , m ∗ , m ∗ , K, L ) only. If additionally P (( Y, Z ) ∈ O ) → 1 , then ξ in the previous display satisfies ξ d - - → N ( 0 , I K ) and

The proof of Theorem S5.2 is presented in Section S5.3.

S5.3 Proof of Theorem S5.2

We first present a few useful lemmas, whose proofs are given at the end of this subsection.

Lemma S5.3 (Proof is given on page 24) . Let z ∼ N ( 0 , σ 2 I n ) and F : R n → R n × K be weakly differentiable with E ∥ F ( z ) ∥ 2 F < ∞ . Let ˜ z be an independent copy of z . Then

Lemma S5.4 (Proof is given on page 25) . If G, ˜ G ∈ R n × K satisfy m ∗ I K ⪯ 1 n G T G ⪯ m ∗ I K and m ∗ I K ⪯ 1 n ˜ G T ˜ G ⪯ m ∗ I K for some positive constants m ∗ and m ∗ . Then

where L 1 , L 2 are positive constants depending on ( K,m ∗ , m ∗ ) only.

Lemma S5.5 (Proof is given on page 26) . Let the assumptions in Theorem S5.2 be fulfilled. Let Y ∈ R n × ( K +1) be fixed. If a minimizer ˆ A ( Y, Z ) exists at Z , then Z ↦→ ˆ A ( Y, Z ) exists and is differentiable in a neighborhood of Z with derivative

where M = [ ∑ H i ⊗ ( z i z T )] -1 . It immediately follows that

Corollary S5.6 (Proof is given on page 26) . Under the same conditions of Lemma S5.5, for G = ∑ n i =1 e i g T i , we have for each i ∈ [ n ] , j ∈ [ p ] ,

Now we are ready to prove Theorem S5.2.

Proof of Theorem S5.2. Let h : O → R n × K be h ( Y, Z ) = G ( Y, Z )( G ( Y, Z ) T G ( Y, Z )) -1 / 2 . In most of this proof, we will omit the dependence ( Y, Z ) on h, ˆ A,g i , H i , G, V to lighten notation. By Lemma S5.4, we know this h is LL 2 n -1 / 2 -Lipschitz in the sense that ∥ h ( Y, Z ) -h ( Y, ˜ Z ) ∥ F ≤ LL 2 n -1 / 2 ∥ Z -˜ Z ∥ F for all { ( Y, Z ) , ( Y, ˜ Z ) } ⊂ O . By Kirszbraun theorem, there exists a function H : R n × ( K +1) × R n × p → R n × K (an extension of h from O to R n × ( K +1) × R n × p ) such that H ( Y, Z ) = h ( Y, Z ) for all ( Y, Z ) ∈ O , ∥ H ( Y, Z ) ∥ op ≤ 1 and ∥ H ( Y, Z ) -H ( Y, ˜ Z ) ∥ F ≤ LL 2 n -1 / 2 ∥ Z -˜ Z ∥ F for all { ( Y, Z ) , ( Y, ˜ Z ) } ⊂ R n × ( K +1) × R n × p .

For each j ∈ [ p ] , let z j = Ze j be the j -th column of Z to distinguish it from the notation z i , which means the i -th row of Z . Let ˇ z ∼ N ( 0 , I n ) be an independent copy of each columns of Z , and ˇ Z j = Z ( I p -e j e T j ) + ˇ z e T j . That is, ˇ Z j replaces the j -th column of Z by ˇ z . By definition, z 1 ⊥ ˇ z and z 1 ⊥ ˇ Z 1 .

Let ξ = -[ H ( Y, ˇ Z 1 )] T z 1 ∈ R K , then ∥ ξ ∥ 2 ≤ ∥ z 1 ∥ since ∥ H ( Y, ˇ Z 1 ) ∥ op ≤ 1 . It follows that

Note that the first p -K columns of Z are exchangeable, because they are i.i.d. and independent of the response Y , we have for each ℓ ∈ [ p -K ] ,

where the last line holds for any ℓ ∈ [ p -K ] because ( z 1 , e T 1 ˆ A, ˇ Z ) d = ( z ℓ , e T ℓ ˆ A, ˇ Z ℓ ) . Therefore,

where Rem ℓ = ∑ n i =1 [ -( g T i ⊗ e T ℓ ) M ( H i ⊗ z i )( G T G ) -1 / 2 + e T i G ∂ ( G T G ) -1 / 2 ∂z iℓ ] from Corollary S5.6, and M = [ ∑ n i =1 ( H i ⊗ z i z T i )] -1 from Lemma S5.5. Using ( a + b ) 2 ≤ 2 a 2 +2 b 2 , the above display can be bounded by sum of two terms, denoted by ( RHS ) 1 and ( RHS ) 2 . For the first term,

Let F ( z ℓ ) = H ( Y, Z ( I -e ℓ e T ℓ ) + z ℓ e T ℓ ) = H ( Y, Z ) , then F (ˇ z ) = H ( Y, ˇ Z ℓ ) . Apply Lemma S5.3 to F ( z ℓ ) conditionally on Z ( I -e ℓ e T ℓ ) , we obtain

where the first equality uses z T ℓ h ( Y, Z ) = 0 from the KKT conditions Z T G = 0 and h ( Y, Z ) = G ( G T G ) -1 / 2 . It follows that

Note that the integrand in the last display is actually the squared Frobenius norm of the Jacobian of the mapping from R n × p to R n × K : Z ↦→ H ( Y, Z ) . This Jacobian is a matrix with nK rows and np columns, has rank at most nK and operator norm at most LL 2 n -1 / 2 because Z ↦→ H ( Y, Z ) is LL 2 n -1 / 2 -Lipschitz from Lemma S5.4. Using ∥ A ∥ 2 F ≤ rank( A ) ∥ A ∥ 2 op , we obtain

For the second term ( RHS ) 2 = 2 p -K ∑ p -K ℓ =1 E [ I { ( Y, Z ) ∈ O }∥ Rem ℓ ∥ 2 ] . By definition of Rem ℓ and ( a + b ) 2 ≤ 2 a 2 +2 b 2 , we obtain

We next bound (S5.3) and (S5.4) one by one. For (S5.3), we focus on the norm without ( G T G ) -1 / 2 which is ∥ ∑ n i =1 ( g T i ⊗ e T ℓ ) M ( H i ⊗ z i ) ∥ . With ∥ a ∥ = max u : ∥ u ∥ =1 a T u in mind, let us multiply to the right by a unit vector u ∈ R K and instead bound

because the rank of the matrix inside the trace is at most K and Tr[ · ] ≤ K ∥ · ∥ op holds. Then

Next, ∥ ∑ i ( H i u ⊗ e i e T i ) ∥ op = ∥ ∑ i ( H i ⊗ e i e T i )( u ⊗ I n ) ∥ op ≤ 1 because H i ⪯ I K and ∥ u ∥ = 1 . In summary, the norm in (S5.3) is bounded from above by

To bound (S5.3), since in the event ( Y, Z ) ∈ O , m ∗ I K ⪯ 1 n G T G ⪯ m ∗ I K and 1 n ∑ n i =1 ( H i ⊗ z i z T i ) ⪰ c ∗ I pK , we have ∥ G ∥ op ≤ √ m ∗ n , hence ∥ M ∥ op ≤ c -1 ∗ . Thus, the above display can be bounded by √

Since Z ∈ R n × p has i.i.d. N (0 , 1) entries, [Davidson and Szarek, 2001, Theorem II.13] implies that E ∥ Z ∥ op ≤ √ n + √ p ≤ 2 √ n. Therefore,

Now we bound (S5.4). Since

Using ∥ G ∥ 2 F ≤ nK , and the mapping Z ↦→ ( G T G ) -1 / 2 is LL 1 n -1 - Lipschitz on O using Lemma S5.4, we conclude that

Combining the above bounds on ( RHS ) 1 and ( RHS ) 2 , we have

where the constant depends on ( c ∗ , K, m ∗ , m ∗ , L ) only because L 1 and L 2 are constants depending on ( K,m ∗ , m ∗ ) only.

If additionally P (( Y, Z ) ∈ O ) → 1 , we have P (( Y, ˇ Z 1 ) ∈ O ) → 1 using ( Y, Z ) d = ( Y, ˇ Z 1 ) . Therefore,

By (S5.5), we know ( G T G ) -1 / 2 V ˆ A T e 1 -ξ d - - → 0 when P (( Y, Z ) ∈ O ) → 1 . Hence, we conclude

We next prove Lemmas S5.3 to S5.5 and corollary S5.6.

Proof of Lemma S5.3. Let z 0 = ( z T , ˜ z T ) T ∈ R 2 n , then z 0 ∼ N ( 0 , σ 2 I 2 n ) . For each k ∈ [ K ] , let f ( k ) : R 2 n → R 2 n be

so that z T 0 f ( k ) ( z 0 ) = z T [ F ( z ) -F (˜ z )] e k , and div f ( k ) ( z 0 ) = ∑ n i =1 ∂e T i F ( z ) e k ∂z i . Applying the second order Stein formula [Bellec and Zhang, 2021] (see also [Tan et al., 2022, Lemma F.1] for a

collection of such formulas) to f ( k ) gives, with Jac denoting the Jacobian,

where the last inequality uses the Gaussian Poincaré inequality, and the Cauchy-Schwarz inequality Tr( A 2 ) ≤ ∥ A ∥ 2 F . Summing over k ∈ [ K ] gives the desired inequality.

Proof of Lemma S5.4. We first prove G ↦→ G T G is Lipschitz by noting

Then we show G T G ↦→ ( G T G ) -1 is Lipschitz. Let A = G T G and ˜ A = ˜ G T ˜ G , we have

We next prove ( G T G ) -1 ↦→ ( G T G ) -1 / 2 is Lipschitz. Let S = ( G T G ) -1 , S ′ = ( ˜ G T ˜ G ) -1 , and if u with ∥ u ∥ = 1 is the eigenvector of √ S -√ ˜ S with eigenvalue d , then

As d can be chosen as ±∥ √ S -√ ˜ S ∥ op (this argument is a special case of the Hemmen-Ando inequality [van Hemmen and Ando, 1980]), this implies

Combining the above Lipschitz results, we have

It immediately follows that

That is, the mapping G ↦→ ( G T G ) -1 / 2 is L 1 n -1 -Lipschitz, where L 1 = √ Km -2 ∗ m ∗ .

For the second statement, the result follows by

Hence,

Proof of Lemma S5.5. Recall the KKT conditions ∑ n l =1 z l g T l = 0 p × K . We look for the derivative with respect to z ij . Denoting derivatives with a dot, we find by the chain rule and product rule

Thus, differentiating the KKT conditions w.r.t. z ij by the product rule gives

That is,

We then move the term involving ˙ A to one side, and vectorize both sides,

With M = [ ∑ n l =1 ( H l ⊗ z l z T l )] -1 , we obtain

Hence, using vec( H l ˙ A ⊤ z l ) = vec( z T l ˙ AH l ) = ( H l ⊗ z T l ) vec( ˙ A ) gives

Thus,

Proof of Corollary S5.6. For each i ∈ [ n ] , j ∈ [ p ] , we have by the product rule

n

S6 Proof of Theorem S3.1

Recall that Theorem S5.1 holds for general loss function L i : R K → R provided that conditions (1) and (2) in Theorem S5.1 hold. In this section, we consider the multinomial logistic loss function L i defined in Section S3.1. To be specific,

In order to apply Theorem S5.1, we need to verify that, when L i in (S6.1) is used, the two conditions (1) and (2) in Theorem S5.1 hold. To this end, we present a few lemmas in the following two subsections, which will be useful for asserting the conditions (1) and (2) when we apply Theorem S5.1 to prove Theorem S3.1.

S6.1 Control of the singular values of the gradients and Hessians

Before stating the lemmas that assert the conditions in Theorem S5.1, define

Lemma S6.1 (deterministic result on gradient) . Let L i be defined as in (S6.1) . Assume that either Assumption 2.2 or Assumption S1.1 holds. If Y ∈ U y , for any M ∈ R n × K such that ∥ MQ T ∥ 2 F ≤ nτ , we have

where m ∗ is a positive constant depending on ( K,γ,τ ) only.

Proof of Lemma S6.1. Without loss of generality, let's assume that γn is an integer. Otherwise, we can replace it with the greatest integer less than or equal to γn , denoted as ⌊ γn ⌋ .

If Y ∈ U y , there exists at least γn many disjoint index sets { S 1 , . . . , S γn } such that the following hold for each l ∈ [ γn ] ,

Since S l are disjoint and ∪ γn l =1 S l ⊂ [ n ] , we have

It follows that at most αn many of l ∈ { 1 , 2 , ..., γn } s.t. ∑ i ∈ S l ∥ QM T e i ∥ 2 > τ/α , otherwise the previous display can not hold. In other words, there exists a subset L ∗ ⊂ { 1 , 2 , ..., γn } with | L ∗ | ≥ ( γ -α ) n s.t. ∑ i ∈ S l ∥ QM T e i ∥ 2 ≤ τ/α for all l ∈ L ∗ . Define the index set I = ∪ l ∈ L ∗ S l , then | I | ≥ ( K +1) n ( γ -α ) , and ∥ QM T e i ∥ ∞ ≤ √ τ/α for all i ∈ I . Let us take α = γ/ 2 , then | L ∗ | ≥ γ 2 n and | I | ≥ γ ( K +1) n/ 2 . Recall that L i ( u ) = L i ( Qu ) , we have ∇ L i ( u ) = Q T ∇ L i ( Qu ) . Thus,

where p i ∈ R K +1 and its k -th entry satisfying

for some constant c ∈ (0 , 1) depending on ( τ, α, K ) only. Therefore,

where A l ∈ R ( K +1) × ( K +1) has K +1 rows { y i -p i : i ∈ S l } . We further note that A l is of the form ( I K +1 -P l ) up to a rearrangement of the columns, where P l ∈ R ( K +1) × ( K +1) is a stochastic matrix with entries of the form

By (S6.2), for each l ∈ L ∗ , the stochastic matrix P l is irreducible and aperiodic and ker( I K +1 -P l ) is the span of the all-ones vector 1 . Therefore,

It follows that

We conclude that the rank of ( I K +1 -P l ) Q is K .

If P denotes the set of matrices { P ∈ R ( K +1) × ( K +1) : stochastic with entries in [ c, 1 -c ] } , and S K -1 = { a ∈ R K : ∥ a ∥ = 1 } . By compactness of P and S K -1 , we obtain

where a ∗ ∈ S K -1 , P ∗ ∈ P , and m ∗ is a positive constant depending on ( K,γ,τ ) only. The first inequality above uses the property λ min ( A + B ) ≥ λ min ( A ) + λ min ( B ) , where A and B are two positive semi-definite matrices.

In other words, 1 n ∑ n i =1 ∇ L i ( M T e i ) ∇ L i ( M T e i ) T ⪰ m ∗ I K . For the upper bound, since ∥ Q ∥ op ≤ 1 by definition of Q and all the entries of ( y i -p i )( y i -p i ) T are between -1 and 1 if Assumption 2.2 or Assumption S1.1 holds, we have

Lemma S6.2 (deterministic result on Hessian) . Let L i be defined as in (S6.1) . For all i ∈ [ n ] , we have ∇ 2 L i ( u ) ⪯ I K for any u ∈ R K and

where c ∗ is a positive constant depending on ( K,r ) only.

Proof of Lemma S6.2. Recall that L i ( u ) = L i ( Qu ) , we have

where ∇ 2 L i ( Qu ) = diag( p i ) -p i p T i and the k -th entry of p i ∈ R K +1 is defined as p ik = exp( e T k Qu ) ∑ K +1 k ′ =1 exp( e T k ′ Qu ) for all i ∈ [ n ] , k ∈ [ K + 1] . Thus, p ik ≤ 1 for all i ∈ [ n ] , k ∈ [ K + 1] . For any vector a ∈ S K -1 , we have

where the last inequality uses ∥ Q ∥ op ≤ 1 and p ik ≤ 1 for any k . Hence, ∇ 2 L i ( u ) ⪯ I K for any i ∈ [ n ] .

Now we prove the lower bound. For any u ∈ R K such that ∥ Qu ∥ ∞ ≤ r , we have

for some constant c depending on ( K,r ) only. For any vector a ∈ S K -1 , let η = Qa ∈ R K +1 , then 1 T η = 0 and

where the last equality uses ∑ K +1 k =1 p ik = 1 , and the inequality follows by ( ∑ k p ik η k ) 2 = ( ∑ k √ p ik √ p ik η k ) 2 ≤ ∑ k p ik ∑ k p ik η 2 k using the Cauchy-Schwarz inequality, and here ' = ' holds if and only if √ p ik ∝ √ p ik η k for each k , which is not true since p ik ∈ [ c, 1 -c ] and 1 T η = 0 .

Let H = { Q T (diag( p ) -pp T ) Q : p ∈ [ c, 1 -c ] K +1 } , then H is compact and

for some a ∗ ∈ S K -1 and H ∗ ∈ H . Therefore,

where c ∗ is a positive constant depending on ( K,r ) only.

S6.2 Lipschitz conditions

We first restate the definitions of following sets,

Lemma S6.3. Assume p/n ≤ δ -1 < 1 -α for some α ∈ (0 , 1) and δ > 1 . Let I = { I ⊂ [ n ] : | I | = ⌈ n (1 -α ) ⌉} and P I = ∑ i ∈ I e i e T i . Define

for some positive constants ϕ ∗ , ϕ ∗ , which depend on ( δ, α ) only. Let U ∗ = { ( Y, X ) ∈ U : Y ∈ U y , X ∈ U x } . Then under Assumptions 2.1, 2.3 and 2.4, and if either Assumption 2.2 or Assumption S1.1 holds, we have

(i) P (( Y, X ) ∈ U ∗ ) → 1 as n, p →∞ .

(ii) Let G be defined in Assumption S5.2. If { ( Y, X ) , ( Y, ˜ X ) } ⊂ U ∗ , we have

where L is a positive constant depending on ( K,γ,τ,α ) only.

Proof of Lemma S6.3. We first prove statement (i). Under Assumption 2.1, [Bellec, 2022, Lemma 7.7] implies

for some positive constant ϕ ∗ depending on ( δ, α ) only. Furthermore, [Davidson and Szarek, 2001, Theorem II.13] implies

for some positive constant ϕ ∗ depending on δ only. Therefore, P ( X ∈ U x ) → 1 . Under Assumption 2.3, we have P ( Y ∈ U y ) → 1 . Under Assumption 2.4, we have P (( Y, X ) ∈ U ) → 1 . In conclusion, under Assumptions 2.1, 2.3 and 2.4, we have P (( Y, X ) ∈ U ∗ ) → 1 as n, p →∞ .

Now we prove the statement (ii). For a fixed Y , let ( Y, X ) , ( Y, ˜ X ) ∈ U ∗ , ˆ B, ˜ B be their corresponding minimizers of (S3.3), and G, ˜ G be their corresponding gradient matrices. We first provide some useful results derived from the KKT conditions. From the KKT conditions X T G = ˜ X T ˜ G = 0 , we have

Since ∥∇ 2 L i ( u ) ∥ op ≤ 1 for any u ∈ R K from Lemma S6.2, ∇ L i ( · ) is 1-Lipschitz. Thus,

If ( Y, X ) , ( Y, ˜ X ) ∈ U , we have ∥ X ˆ BQ T ∥ 2 F + ∥ ˜ X ˜ BQ T ∥ 2 F ≤ 2 nτ . That is,

Define the index set then we have | I | ≥ (1 -α ) n by Markov's inequality. Thus, for all i ∈ I , we have ∥ Q ˆ B T x i ∥ ∞ ∨ ∥ Q ˜ B T ˜ x i ∥ ∞ ≤ √ 2 τ α .

Applying Lemma S6.2 with r = √ 2 τ α gives

where c ∗ is a constant depending on ( K,τ,α ) . Therefore,

We next bound the first line from below by expanding the squares,

where in the last inequality we use the constant ϕ ∗ in (S6.3). Therefore, we obtain

Together with the inequality that ∥ G -˜ G ∥ 2 F ≤ ⟨ X ˆ B -˜ X ˜ B,G -˜ G ⟩ , we obtain

where we bound ⟨ ˜ X ( ˆ B -˜ B ) , P I ( X -˜ X ) ˆ B ⟩ by definition of ϕ ∗ ,

Now we derive a bound of the form ∥ Σ 1 / 2 ˆ B ∥ F ≲ ∥ G ∥ F / √ n . To this end, since ϕ ∗ ∥ Σ 1 / 2 ˆ B ∥ F ≤ ∥ P I X ˆ B ∥ F / √ n ≤ ∥ X ˆ B ∥ F / √ n = ∥ X ˆ BQ T ∥ F / √ n ≤ √ τ .

Applying Lemma S6.1 to M = X ˆ B , we have 1 n ∑ n i =1 g i g T i ⪰ m ∗ I K . Therefore,

In conclusion, if { ( Y, X ) , ( Y, ˜ X ) } ⊂ U ∗ then

√ n ∥ Σ 1 / 2 ( ˆ B -˜ B ) ∥ F + ∥ G -˜ G ∥ F ≤ Cn -1 / 2 ∥ ( X -˜ X )Σ -1 / 2 ∥ op ∥ G ∥ F ≤ CK ∥ ( X -˜ X )Σ -1 / 2 ∥ op , where C is a constant depending on ( K,γ,τ,α ) only. Note that ∥ G ∥ F ≤ √ nK since all entries of G are in [ -1 , 1] .

Lemma S6.4. If p/n ≤ δ -1 < (1 -α ) for some α ∈ (0 , 1) and δ > 1 . If ( Y, X ) ∈ U and X ∈ U x , where U x is defined in Lemma S6.3, we have

where c 1 is a positive constant depending on ( K,τ,α,ϕ ∗ ) only.

Proof of Lemma S6.4. If ( Y, X ) ∈ U , we have ∥ X ˆ BQ T ∥ 2 F ≤ nτ . Define the index set ,

then we have | I | ≥ (1 -α ) n by Markov's inequality. Therefore, for any i ∈ I , ∥ Q ˆ B T x i ∥ ∞ ≤ τ α . Applying Lemma S6.2 with u = ˆ B T x i and r = τ α , we have for any i ∈ I , H i = ∇ 2 L i ( ˆ B T x i ) ⪰ c ∗ I K for some positive constant c ∗ depending on ( K,τ,α ) only. Therefore, if ( Y, X ) ∈ U and X ∈ U x , we have

This implies that

where P I = ∑ i ∈ I e i e T i and c 1 is a positive constant depending on ( K,τ,α,ϕ ∗ ) only.

S6.3 Proof of Theorem S3.1

The proof of Theorem S3.1 is a direct consequence of Theorem S5.1 by noting

which is a consequence of the identities G = ∑ n i =1 e i g T i and V = ∑ n i =1 V i .

It thus remains to verify the conditions (1) and (2) in Theorem S5.1 from the assumptions in Theorem S3.1.

Applying Lemma S6.3 with α chosen as 1 -δ -1 / 2 , we have for { ( Y, X ) , ( Y, ˜ X ) } ⊂ U ∗ ,

where L is a positive constant depending on ( K,γ,τ,δ ) only.

Apply Lemma S6.4 with the same α = 1 -δ -1 / 2 , we have for ( Y, X ) ∈ U ∗ ,

where c ∗ is a positive constant depending on ( K,τ,δ ) only.

Applying Lemma S6.1 with M = X ˆ B , we have for ( Y, X ) ∈ U ∗ ,

where m ∗ is a positive constant depending on ( K,γ,τ ) only. Therefore, the conditions (1) and (2) in Theorem S5.1 hold when the multinomial logistic loss is used. This completes the proof of Theorem S3.1.

S7 Other proof

S7.1 Proof of Equation (1.12) (Classical asymptotic theory with fixed p )

Here we provide a derivation of the asymptotic distribution of MLE under classical setting, where p is fixed and n tends to infinity.

We first calculate the Fisher information matrix of the multinomial logistic log-odds model (1.8) with covariate x ∼ N ( 0 , Σ) and response y ∈ { 0 , 1 } K +1 one-hot encoded satisfying ∑ K +1 k =1 y k = 1 . Note that the model (1.8) can be rewritten as

The likelihood function of a parameter A ∈ R p × K is

where we used ∑ K +1 k =1 y k = 1 . Thus, the log-likelihood function is

It is more convenient to calculate the Fisher information matrix on the vector space R pK instead of the matrix space R p × K . To this end, let θ = vec( A T ) , then x T Ae k = e T k A T x = ( x T ⊗ e T k ) vec( A T ) = ( x T ⊗ e T k ) θ , and the log-likelihood function parameterized by θ is

By multivariate calculus, we obtain the Fisher information matrix evaluated at θ ∗ = vec( A ∗ T ) ,

where π ∗ ∈ R K with k -th entry π ∗ k = exp( x T A ∗ e k ) 1+ ∑ K k ′ =1 exp( x T A ∗ e k ′ ) for each k ∈ [ K ] .

From classical maximum likelihood theory, for instance [Van der Vaart, 1998, Chapter 5], we have

where ˆ θ = vec( ˆ A T ) and ˆ A is the MLE of A ∗ . Furthermore, if the j -th covariate is independent of the response, we know e T j A ∗ = 0 T , then

where

holds by the independence between the j -th covariate and the response under H 0 in (1.10). This completes the proof of Equation (1.12).

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper aims to study the statistics of the multinomial logistic MLE on null covariates in a multiclass classification setting in the asymptotic proportional regime, i.e., in the limit in which the size of the dataset $n$ and of the dimension of the covariates $p$ are both sent to infinity but with finite ratio. The work extends previous results by Sur and Candès in which the normality of the MLE on null covariates was proven in the case of binary classification and introduces a proper statistic for a $\\chi^2$ test on the relevance of a feature.\n\nSTRENGTHS:\nThe paper is clearly and carefully written, the validity conditions of the results are clearly stated and the outcome of the theoretical analysis is supported by robust numerical evidence. The characterization of the statistics of the MLE on null covariates in the case of multiclass classification in the high dimensional limit is timely. The authors also compare the high-d and the classical theory, showing the remarkable difference in the provided predictions. Finally, they also test the validity of their results beyond the Gaussian hypothesis adopted for the derivation of their theorem.\n\nWEAKNESSES:\nA possible theoretical weakness of the paper is the fact that the existence of the MLE is assumed (Assumption 2.4) and not fully characterized. This \"weakness\" is acknowledged by the authors themselves in the final section of the main text.\n\nQUESTIONS:\nOut of curiosity, the authors state that they expect their results to hold for covariates with distribution having \"sufficiently light\" tails. Could they comment on this? Do they expect, for example, sub-gaussianity to be sufficient for their results to hold? \n\nAs a (very) minor remark, Theorem 2.1 and Theorem 2.2 exhibit the quantities $\\mathsf y_i-\\hat{\\mathsf p}_i\\equiv -\\mathsf g_i$, but a different notation is used. The analogy of the two results might appear more evident at first sight using the same notation in both theorems (eg replacing $-\\mathsf y_i+\\hat{\\mathsf p}_i$ with $\\mathsf g_i$ in the first theorem).\n\nLIMITATIONS:\nThe work is of theoretical nature and all limitations are within the well specified hypotheses of the reported theorems.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the asymptotic distribution of multinomial logistic regression in the high dimensions. Under the assumption of a Gaussian design (and a few other assumptions), it establishes and characterizes the asymptotic normality of the coefficient of a null feature. This extends previous results for high-dimensional binary logistic regression to $K \\geq 3$ number of classes. This result enables proper significant testing in high-dimensions, for which the classical fixed-$p$ asymptotic fails to control the type-I error.\n\nSTRENGTHS:\n1. The paper extends previous results for high-dimensional logistic regression to the multi-class setting. \n2. The paper is well-written and the technical content is presented rigorously. \n3. The simulations results seem to match the asymptotic theory very well.\n\nWEAKNESSES:\n1. The hypothesis $H_0$ seems stronger than the hypothesis $H_0'$: the population coefficient (in terms of KL projection) for feature $j$ is zero. Supposedly $H_0'$ reduces to $H_0$ when the model is well-specified. In terms of the data generating mechanism given by Assumption 2.1, it seems possible that when $y_i = f(U_i, x_i^{T} B^{\\ast})$ does not hold, a feature that does not satisfy $H_0$ may still have zero as its population \"true\" coefficient. I am wondering whether $H_0$ can be suitably weakened.\n2. The result relies on a random Gaussian design, although it is suggested that the result probably extends to other random designs with light tails. \n3. The paper does not characterize the distribution for non-null features, so the study of parameter inference for high-dimensional multinomial logistic regression is still incomplete.\n\nQUESTIONS:\n1. Per my first comment on the \"Weaknesses\", can authors comment on the extent to which $H_0$ can be weakened? \n2. To what extent is the \"universality\" expected to hold for this problem? Does the asymptotic normality fail under a random, heavy-tailed design? What about fixed designs?\n\nLIMITATIONS:\nI do not foresee any potential negative societal impact of this work. In terms of limitations, I think obtaining results for the non-null features would make this work a much stronger paper.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors provide asymptotic characterization of the behavior of the maximum likelihood estimator (MLE) of multinomial logistic model (with more than two classes), in the high-dimensional regime where the dimension and the sample size of data go to infinity at the same rate.\n\nUnder some technical assumptions (that may need some further elaborations), this paper develops asymptotic normality and asymptotic chi-square results for the multinomial logistic MLE on null covariates (see Theorem 2.1 and 2.2). The proposed results can be used for statistical inference and significance test of some specific features. Numerical experiments on synthetic data are provided in Section 3 to validate the proposed theory.\n\nSTRENGTHS:\nThis papers focuses on the fundamental and important problem of MLE of multinomial logistic model in the modern high-dimensional regime.\nThe proposed theory improves prior art in characterizing the asymptotic normality and asymptotic chi-square results, both of significance to statistics and ML. \nThe paper is in general well written and easy to follow.\n\nWEAKNESSES:\nI do not have strong concerns to raise for this paper. \nSee below for some detailed comments and/or questions.\n\nQUESTIONS:\n1. While I am almost fine with Assumption 2.1 and 2.2 (just being curious, is some upper bound on the spectral norm of the covariance $\\Sigma$ needed? or it is just a matter of scaling with respect to $n,p$?), I am a bit confused by Assumption 2.3 and 2.4: are they something intrinsic or for the ease of technical analysis? What happens if, say Assumption 2.3 is violated? Can we have something similar but just more involved or the MLE is totally different? Also, Assumption 2.4 is a bit misleading, in the sense that the assumption is not instinct, and should perhaps be reduced into some assumption on the dimension ratio $p/n$ and/or statistics of the data? I believe it makes more sense to assume something like \"the dimension ratio $p/n$, covariance $\\Sigma$ and xxx satisfy that xxx\". I am also confused by the paragraph after Assumption 2.4 and I am not sure the convergence of some multinomial regression solver can be used as a rigorous theoretical indicator. The algorithm may converge (or believed to converge) due to many reasons. Perhaps some better (numerical) criterion can be proposed by, e.g., checking the gradient and/or Hessian of the point of interest.\n2. for the sake of presentation and use, it be helpful to present Theorem 2.2 and the estimation of $\\Omega_{jj}$ in form of an algorithm.\n3. Almost nothing is mentioned for the proof of the theoretical results (Theorem 2.1 and 2.2): is the proof technically challenging or contains some ingredients and/or intermediate results that may be of independent interest? Could the authors elaborate more on this?\n\nLIMITATIONS:\nThis paper is primarily of a theoretical nature, and I do not see any potential negative societal impact of this work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the asymptotic distribution of the MLE of the multinomial logistic regression model when the sample size and the number of parameters are of the same order.  The validity of the asymptotic theories is evaluated through extensive simulation studies. The paper is overall very well written and the presentation is clear.\n\nSTRENGTHS:\nThe paper is very well written, and the problem under consideration is of great interest. Theoretical results are sound and the numerical experiments are sufficient.\n\nWEAKNESSES:\nOne of the motivating example of the paper is to study classification with 3 or more classes. And one of the most important goal in classification is the classification error or AUC. In my opinion, the impact of the proposed method on classification errors should be evaluated through simulation studies and some real data examples.\n\nQUESTIONS:\n1. Related to my question in the weakness section, one could use the proposed tests to exclude some noise covariates in the multinomial regression and see how much improvements can be achieved in terms of classification errors. Preferably, one should compare its performance to some existing methods. \n\n2. I would strongly suggests applying the proposed methodology to some real-world benchmark data set. The statistical tests may provide additional insights and the classification errors can be evaluated through cross-validation.\n\n3. The paper's theoretical framework appears to heavily rely on the seminal work of Sur and Candes (2019), because of which I still have some reservation on the novelty of the theory. To address concerns about the novelty of the theory, it would be wise to explicitly highlight the unique theoretical challenges posed by the multinomial logistic regression model compared to the more commonly studied binary logistic regression model.\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}