SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Pierre C Bellec. Observable adjustments in single-index models for regularized m-estimators. arXiv preprint arXiv:2204.06990, 2022.
Pierre C. Bellec and Cun-Hui Zhang. Second-order Stein: SURE for SURE and other applications in high-dimensional inference. Ann. Statist., 49(4):1864-1903, 2021. ISSN 0090-5364.
Raphael Berthier, Andrea Montanari, and Phan-Minh Nguyen. State evolution for approximate message passing with non-separable functions. Information and Inference: A Journal of the IMA, 9(1):33-79, 2020.
Emmanuel J. Candès and Pragya Sur. The phase transition for the existence of the maximum likelihood estimate in high-dimensional logistic regression. The Annals of Statistics, 48(1):27 - 42, 2020.
Elisabetta Cornacchia, Francesca Mignacco, Rodrigo Veiga, Cédric Gerbelot, Bruno Loureiro, and Lenka Zdeborova. Learning curves for the multi-class teacher-student perceptron. arXiv preprint arXiv:2203.12094, 2022.
Jan Salomon Cramer. The origins of logistic regression. Tinbergen Institute Working Paper, 2002.
Yatin Dandi, Ludovic Stephan, Florent Krzakala, Bruno Loureiro, and Lenka Zdeborova. Universality laws for gaussian mixtures in generalized linear models. arXiv preprint arXiv:2302.08933, 2023.
Supplementary Material of "Multinomial Logistic Regression: Asymptotic Normality on Null Covariates in High-Dimensions"
Let us define some standard notation that will be used in the rest of this supplement. For a vector  $v \in \mathbb{R}^n$ , let  $\| v \|_{\infty} = \max_{i \in [n]} |v_i|$  denote the infinity norm of vector  $v$ . If  $A$  is symmetric, we define  $\lambda_{\min}(A)$  and  $\| A \|_{op}$  as the minimal and maximal eigenvalues of  $A$ , respectively. For two symmetric matrices  $A, B$  of the same size, we write  $A \preceq B$  if and only if  $B - A$  is positive semi-definite.
Diagram: Organization of the proofs
The following diagram summarizes the different theorems and lemmas, and the relationships between them.
S1 Extension:  $q$  repeated measurements
Let integer  $q \geq 1$  be a constant independent of  $n, p$ . Our results readily extend if  $q$  labels are observed for each observed feature vector  $x_{i}$ , and the corresponding  $q$  one-hot encoded vectors are averaged into  $y_{i} \in \{0, \frac{1}{q}, \frac{2}{q}, \dots, 1\}^{K+1}$ . Concretely, for each observation  $i \in [n]$ ,  $q$  i.i.d. labels  $(Y_{i}^{m})_{m \in [q]}$  are observed with each  $Y_{i}^{m} \in \{0, 1\}^{K+1}$  one-hot encoded and  $y_{ik} = \frac{1}{q} \sum_{m=1}^{q} Y_{ik}^{m}$ , for instance in a repeated multinomial regression model with  $\mathbb{P}(Y_{ik}^{m} = 1 | x_{i})$  equal to right-hand side of (1.4). In this case where  $(Y_{i}^{m})_{m \in [q]}$  are i.i.d., Assumption 2.3 is satisfied by the law of large numbers if  $\min_{k \in [K+1]} \mathbb{P}(Y_{ik}^{m} = 1) > 0$  since  $q$  is constant. For this  $q$  repeated measurements model, the negative log-likelihood function of a parameter  $\mathsf{B} \in \mathbb{R}^{p \times (K+1)}$  is
$$
\begin{array}{l} - \sum_ {i = 1} ^ {n} \sum_ {m = 1} ^ {q} \sum_ {k = 1} ^ {K + 1} Y _ {i k} ^ {m} \left[ x _ {i} ^ {T} \mathsf {B e} _ {k} - \log \sum_ {k ^ {\prime} = 1} ^ {K + 1} \exp (x _ {i} ^ {T} \mathsf {B e} _ {k ^ {\prime}}) \right] \\ = q \sum_ {i = 1} ^ {n} \sum_ {k = 1} ^ {K + 1} \mathcal {y} _ {i k} \left[ - x _ {i} ^ {T} \mathsf {B e} _ {k} + \log \sum_ {k ^ {\prime} = 1} ^ {K + 1} \exp \left(x _ {i} ^ {T} \mathsf {B e} _ {k ^ {\prime}}\right) \right] \\ = q \sum_ {i = 1} ^ {n} \left[ \sum_ {k = 1} ^ {K + 1} - y _ {i k} x _ {i} ^ {T} \mathsf {B} e _ {k} + \log \sum_ {k ^ {\prime} = 1} ^ {K + 1} \exp \left(x _ {i} ^ {T} \mathsf {B} e _ {k ^ {\prime}}\right) \right] \\ = q \sum_ {i = 1} ^ {n} \mathsf {L} _ {i} (\mathsf {B} ^ {T} x _ {i}), \\ \end{array}
$$
where the first equality uses  $y_{ik} = \frac{1}{q}\sum_{m=1}^{q}Y_{ik}^{m}$ , the second equality uses  $\sum_{k=1}^{K+1}y_{ik} = 1$  under the following Assumption S1.1, and the last equality uses the definition of  $L_i$  after (1.5).
Assumption S1.1. For all  $i \in [n]$ , the response  $y_i$  is in  $\{0, 1/q, 2/q, \dots, 1\}^{K+1}$  with  $\sum_{k=1}^{K+1} y_{ik} = 1$ .
In such repeated measurements model, we replace Assumption 2.2 with Assumption S1.1 under which the following Theorem S1.1 holds.
Theorem S1.1. Let  $q \geq 2$  be constant. Let Assumptions S1.1, 2.1, 2.3 and 2.4 be fulfilled. For any  $j \in [p]$  such that  $H_0$  in (1.10) holds, we have the convergence in distribution (2.1), (2.2) and (2.3).
Proof of Theorem S1.1. Under the assumptions in Theorem S1.1, the MLE  $\hat{\mathbf{B}}$  for this  $q$  repeated measurements model is the minimizer of the optimization problem
$$
\hat {\mathsf {B}} \in \operatorname * {a r g   m i n} _ {\mathsf {B} \in \mathbb {R} ^ {p \times (K + 1)}} \sum_ {i = 1} ^ {n} \mathsf {L} _ {i} (\mathsf {B} ^ {T} x _ {i})
$$
as in (1.5). Similar to the non-repeated model, the MLE  $\hat{A}$  for the identifiable log-odds model can be expressed as
$$
\hat {A} = \operatorname * {a r g   m i n} _ {A \in \mathbb {R} ^ {p \times K}} \sum_ {i = 1} ^ {n} \mathrm {L} _ {i} ((A, \mathbf {0} _ {p}) ^ {T} x _ {i}).
$$
The only difference between this  $q$  repeated measurements model and the non-repeated model considered in the main text is that the response  $y_{ik}$  for this  $q$  repeated measurements model is now valued in  $\{0,1/q,2/q,\dots,1\}$ . Because the proofs of Theorems 2.1 and 2.2 do not require the value of  $y_{ik}$  to be  $\{0,1\}$ -valued. Theorem S1.1 can be proved by the same arguments used in the proof of Theorems 2.1 and 2.2.
S2 Implementation details and additional figures
The pivotal quantities in our main results Theorems 2.1 and 2.2 involve only observable quantities that can be computed from the data  $(x_{i},y_{i})_{i\in [n]}$ . In this section we provide an efficient way of computing the matrix  $\mathbf{V}_i$  appearing in Theorems 2.1 and 2.2.
Fast computation of  $\mathsf{V}_i$ . Recall the definition of  $\mathsf{V}_i$  in Theorem 2.1,
$$
\mathsf {V} _ {i} = \mathsf {H} _ {i} - \left(\mathsf {H} _ {i} \otimes x _ {i} ^ {T}\right) \left[ \sum_ {l = 1} ^ {n} \mathsf {H} _ {l} \otimes \left(x _ {l} x _ {l} ^ {T}\right) \right] ^ {\dagger} \left(\mathsf {H} _ {i} \otimes x _ {i}\right).
$$
The majority of computational cost in calculating  $\mathsf{V}_i$  lies in the step of calculating its second term
$$
\left(\mathsf {H} _ {i} \otimes x _ {i} ^ {T}\right) \left[ \sum_ {l = 1} ^ {n} \mathsf {H} _ {l} \otimes \left(x _ {l} x _ {l} ^ {T}\right) \right] ^ {\dagger} \left(\mathsf {H} _ {i} \otimes x _ {i} ^ {T}\right).
$$
Here we provide an efficient way to compute this term using the Woodbury matrix identity. Since  $\mathsf{H}_i\mathbf{1}_{K + 1} = \mathbf{0}_{K + 1}$ , we have  $\ker (\mathsf{H}_i\otimes (x_ix_i^T))$  is the span of  $\{\mathbf{1}_{K + 1}\otimes e_j:j\in [p]\}$ , where  $\mathbf{1}_{K + 1}$  is the all-ones vector in  $\mathbb{R}^{K + 1}$ . Therefore, the second term in  $\mathsf{V}_i$  can be rewritten as
$$
\begin{array}{l} \left(\mathsf {H} _ {i} \otimes x _ {i} ^ {T}\right) \left[ \sum_ {l = 1} ^ {n} \mathsf {H} _ {l} \otimes \left(x _ {l} x _ {l} ^ {T}\right) \right] ^ {\dagger} \left(\mathsf {H} _ {i} \otimes x _ {i}\right) \\ = \left(\mathsf {H} _ {i} \otimes x _ {i} ^ {T}\right) \left[ \sum_ {l = 1} ^ {n} \mathsf {H} _ {l} \otimes \left(x _ {l} x _ {l} ^ {T}\right) - \sum_ {j = 1} ^ {p} (\mathbf {1} \otimes e _ {j}) (\mathbf {1} \otimes e _ {j}) ^ {T} \right] ^ {- 1} \left(\mathsf {H} _ {i} \otimes x _ {i}\right). \\ \end{array}
$$
We now apply the Woodbury matrix identity to compute the matrix inversion in the above display. Recall  $\mathsf{H}_i = \mathrm{diag}(\hat{\mathsf{p}}_i) - \hat{\mathsf{p}}_i\hat{\mathsf{p}}_i^T$  , we have
$$
\sum_ {i = 1} ^ {n} \mathsf {H} _ {i} \otimes (x _ {i} x _ {i} ^ {T}) = \sum_ {k = 1} ^ {K + 1} \left(e _ {k} e _ {k} ^ {T}\right) \otimes \left(\sum_ {i = 1} ^ {n} \hat {\mathsf {p}} _ {i k} x _ {i} x _ {i} ^ {T}\right) - \sum_ {i = 1} ^ {n} (\hat {\mathsf {p}} _ {i} \otimes x _ {i}) (\hat {\mathsf {p}} _ {i} \otimes x _ {i}) ^ {T}.
$$
Let  $A = \sum_{k=1}^{K+1}(e_k e_k^T) \otimes (\sum_{i=1}^n \hat{\mathfrak{p}}_{ik} x_i x_i^T)$ , and  $U \in \mathbb{R}^{p(K+1) \times (n+p)}$  with the first  $n$  columns being  $(\hat{\mathfrak{p}}_i \otimes x_i)_{i \in [n]}$  and the following  $p$  columns  $(\mathbf{1} \otimes e_j)_{j \in [p]}$ . Then the term we want to invert is  $A - UU^T$ , where  $A$  is a block diagonal matrix and can be inverted by inverting each block separately. By the Woodbury matrix identity, we have
$$
\left(A - U U ^ {T}\right) ^ {- 1} = A ^ {- 1} - A ^ {- 1} U \left(- I _ {n + p} + U ^ {T} A ^ {- 1} U\right) ^ {- 1} U ^ {T} A ^ {- 1}.
$$
The gain of using the above formula is significant for large  $K$ : instead of inverting the  $p(K + 1) \times p(K + 1)$  matrix  $\sum_{l=1}^{n} \mathsf{H}_l \otimes (x_l x_l^T)$  in the left-hand side, the right-hand side only requires to invert a block diagonal matrix  $A$  and a  $(n + p) \times (n + p)$  matrix  $-I_{n+p} + U^T A^{-1} U$ .
(c)  $(n,K) = (6000,4)$
Figure S1: The upper row: Q-Q plots of the test statistics from (2.3) (in blue) and (1.13) (in orange) for different  $(n, K)$  and  $p = 1000$  using  $\hat{\Omega}_{jj}$ . The lower row: histograms of p-values from classical test and our test for different  $(n, K)$  and  $p = 1000$  using  $\hat{\Omega}_{jj}$ .
(a)  $(n,K) = (4000,2)$
Figure S2: Q-Q plots of the test statistics from (2.3) (in blue) and (1.13) (in orange) for different  $(n, K)$  and  $p = 1000$  using  $\hat{\Omega}_{jj}$ . The upper row: covariates are sampled from Rademacher distribution. The lower row: covariates are sampled from distribution of SNPs.
(c)  $(n,K) = (6000,4)$
(a)  $q = 1$
(b)  $q = 2$
(c)  $q = 4$
Figure S3: Scatter plot of pairs  $(\sqrt{n}\hat{A}_{j1},\sqrt{n}\hat{A}_{j2})$  with the same data generating process as in Figure 1 (a) except using different  $q$ .
S3 Proof of Theorem 2.1
Before proving Theorem 2.1, we present another parametrization of the multinomial logistic regression model. The asymptotic theory of MLE for this new parametrized multinomial logistic model will be used to prove Theorem 2.1.
S3.1 Another parametrization of multinomial logistic regression
Recall the symbol “ $\in$ ” is used in (1.5) to emphasize that the minimizer  $\hat{\mathsf{B}}$  in (1.5) is not unique: if  $\hat{\mathsf{B}}$  is a minimizer of (1.5) then  $\hat{\mathsf{B}} - b\mathbf{1}_{K + 1}^T$  is also a minimizer of (1.5), for any  $b \in \mathbb{R}^p$  and the all-ones vector  $\mathbf{1}_{K + 1}$  in  $\mathbb{R}^{K + 1}$ . Besides the log-odds model (1.8), here we consider another identifiable parametrization of multinomial logistic regression, whose unknown parameter, denoted by  $B^*$ , is in  $\mathbb{R}^{p \times K}$ .
Orthogonal complement. To obtain an identifiable multinomial logistic regression model from (1.4), we consider the symmetric constraint  $\mathsf{B}^{*}\mathbf{1} = \sum_{k = 1}^{K + 1}\mathsf{B}^{*}e_{k} = \mathbf{0}$  as in [Zhu and Hastie, 2004], thus  $\mathsf{B}^{*} = \mathsf{B}^{*}(I_{K + 1} - \frac{\mathbf{11}^{T}}{K + 1})$ , where  $\mathbf{1}$  is the all-ones vector in  $\mathbb{R}^{K + 1}$ . Let  $Q\in \mathbb{R}^{(K + 1)\times K}$  be any matrix such that
$$
I _ {K + 1} - \frac {1}{K + 1} \mathbf {1} \mathbf {1} ^ {T} = Q Q ^ {T}, \quad Q ^ {T} Q = I _ {K}. \tag {S3.1}
$$
We fix one choice of  $Q$  satisfying (S3.1) throughout this supplement. Let  $B^{*} = \mathsf{B}^{*}Q$ , then  $\mathsf{B}^{*} = B^{*}Q^{T}$  and the model (1.4) can be parameterized using  $B^{*}$  as
$$
\mathbb {P} \left(\mathrm {y} _ {i k} = 1 \mid x _ {i}\right) = \frac {\exp \left(x _ {i} ^ {T} B ^ {*} Q e _ {k}\right)}{\sum_ {k ^ {\prime} = 1} ^ {K + 1} \exp \left(x _ {i} ^ {T} B ^ {*} Q e _ {k ^ {\prime}}\right)}, \quad k \in \{1, 2, \dots , K + 1 \}. \tag {S3.2}
$$
The multinomial logistic MLE of  $B^{*}$  in (S3.2) is given by
$$
\hat {B} = \arg \min  _ {B \in \mathbb {R} ^ {p \times K}} \sum_ {i = 1} ^ {n} L _ {i} \left(B ^ {T} x _ {i}\right), \tag {S3.3}
$$
where  $L_{i}:\mathbb{R}^{K}\to \mathbb{R}$  is defined by  $L_{i}(u) = \mathsf{L}_{i}(Qu)$  for all  $u\in \mathbb{R}^K$ . By this construction, we have  $\hat{B} = \hat{\mathsf{B}} Q$  for any minimizer  $\hat{\mathsf{B}}$  of (1.5). Furthermore, by the chain rule using the expressions (1.7), the gradient and Hessian of  $L_{i}$  evaluated at  $\hat{B}^T x_i$  are
$$
g _ {i} := \nabla L _ {i} \left(\hat {B} ^ {T} x _ {i}\right) = Q ^ {T} \mathbf {g} _ {i}, \quad H _ {i} := \nabla^ {2} L _ {i} \left(\hat {B} ^ {T} x _ {i}\right) = Q ^ {T} \mathbf {H} _ {i} Q. \tag {S3.4}
$$
Throughout, we use serial upright letters to denote quantities defined on the unidentifiable parameter space  $\mathbb{R}^{p\times (K + 1)}$ :
$$
\mathsf {B} ^ {*}, \hat {\mathsf {B}} \in \mathbb {R} ^ {p \times (K + 1)}, \quad \mathsf {L} _ {i}: \mathbb {R} ^ {K + 1} \to \mathbb {R}, \qquad \hat {\mathsf {p}} _ {i}, \mathsf {y} _ {i}, \mathbf {g} _ {i} \in \mathbb {R} ^ {K + 1}, \qquad \mathsf {H} _ {i} \in \mathbb {R} ^ {(K + 1) \times (K + 1)}
$$
and the normal italic font to denote analogous quantities for the identifiable parameter space  $\mathbb{R}^{p\times K}$ :
$$
B ^ {*}, \hat {B} \in \mathbb {R} ^ {p \times K}, \qquad L _ {i}: \mathbb {R} ^ {K} \to \mathbb {R}, \qquad g _ {i} \in \mathbb {R} ^ {K}, \qquad H _ {i} \in \mathbb {R} ^ {K \times K}.
$$
Theorem S3.1 provides the asymptotic normality and the chi-square approximation of null MLE coordinates in high-dimensions where  $n, p \to \infty$  with the ratio  $n / p$  converging to a finite limit.
Theorem S3.1 (Proof is given on page 27). Let Assumptions 2.1, 2.3 and 2.4 be fulfilled. Assume that either Assumption 2.2 or Assumption S1.1 holds. Then for any  $j \in [p]$  such that  $H_0$  in (1.10) holds,
$$
\sqrt {n} \Omega_ {j j} ^ {- 1 / 2} \left(\frac {1}{n} \sum_ {i = 1} ^ {n} g _ {i} g _ {i} ^ {T}\right) ^ {- 1 / 2} \left(\frac {1}{n} \sum_ {i = 1} ^ {n} V _ {i}\right) \hat {B} ^ {T} e _ {j} \xrightarrow {\mathrm {d}} N (0, I _ {K}), \tag {S3.5}
$$
where  $V_{i} = H_{i} - (H_{i}\otimes x_{i}^{T})[\sum_{l = 1}^{n}H_{l}\otimes (x_{l}x_{l}^{T})]^{-1}(H_{i}\otimes x_{i})$
A direct consequence of (S3.5) is the  $\chi^2$  result,
$$
\left\| \sqrt {n} \Omega_ {j j} ^ {- 1 / 2} \left(\frac {1}{n} \sum_ {i = 1} ^ {n} g _ {i} g _ {i} ^ {T}\right) ^ {- 1 / 2} \left(\frac {1}{n} \sum_ {i = 1} ^ {n} V _ {i}\right) \hat {B} ^ {T} e _ {j} \right\| ^ {2} \xrightarrow {\mathrm {d}} \chi_ {K} ^ {2}. \tag {S3.6}
$$
The proof of Theorem S3.1 is deferred to Section S5 and Section S6. In the next subsection, we prove Theorem 2.1 using Theorem S3.1.
S3.2 Proof of Theorem 2.1
We restate Theorem 2.1 for convenience.
Theorem 2.1. Let Assumptions 2.1 to 2.4 be fulfilled. Then for any  $j \in [p]$  such that  $H_0$  in (1.10) holds, and any minimizer  $\hat{\mathbf{B}}$  of (1.5), we have
$$
\underbrace {\sqrt {\frac {n}{\Omega_ {j j}}}} _ {\text {s c a l a r}} \left(\underbrace {\left(\frac {1}{n} \sum_ {i = 1} ^ {n} \mathbf {g} _ {i} \mathbf {g} _ {i} ^ {T}\right) ^ {1 / 2}} _ {\mathbb {R} ^ {(K + 1) \times (K + 1)}}\right) ^ {\dagger} \underbrace {\left(\frac {1}{n} \sum_ {i = 1} ^ {n} \mathbf {V} _ {i}\right)} _ {\mathbb {R} ^ {(K + 1) \times (K + 1)}} \underbrace {\hat {\mathbf {B}} ^ {T} e _ {j}} _ {\mathbb {R} ^ {K + 1}} \xrightarrow {\mathrm {d}} N \left(\mathbf {0}, \underbrace {I _ {K + 1} - \frac {1 1 ^ {T}}{K + 1}} _ {\text {c o v .} \mathbb {R} ^ {(K + 1) \times (K + 1)}}\right), \tag {2.1}
$$
where  $\mathbf{g}_i = -\mathbf{y}_i + \hat{\mathbf{p}}_i$  as in (1.7) and  $\mathsf{V}_i = \mathsf{H}_i - (\mathsf{H}_i \otimes x_i^T)[\sum_{l=1}^{n} \mathsf{H}_l \otimes (x_l x_l^T)]^\dagger (\mathsf{H}_i \otimes x_i)$ .
The proof of Theorem 2.1 is a consequence of Theorem S3.1. To begin with, we state the following useful lemma.
Lemma S3.2. For  $\mathsf{V}_i$  and  $V_{i}$  defined in Theorems 2.1 and S3.1, we have  $V_{i} = Q^{T}\mathsf{V}_{i}Q$
Proof of Lemma S3.2. Since  $H_{i} = Q^{T}\mathsf{H}_{i}Q$ , we have
$$
\begin{array}{l} V _ {i} = H _ {i} - \left(H _ {i} \otimes x _ {i} ^ {T}\right) \left[ \sum_ {i = 1} ^ {n} H _ {i} \otimes \left(x _ {i} x _ {i} ^ {T}\right) \right] ^ {- 1} \left(H _ {i} \otimes x _ {i}\right) \\ = H _ {i} - \left[ \left(Q ^ {T} \mathsf {H} _ {i} Q\right) \otimes x _ {i} ^ {T} \right] \left[ \sum_ {i = 1} ^ {n} \left(Q ^ {T} \mathsf {H} _ {i} Q\right) \otimes \left(x _ {i} x _ {i} ^ {T}\right) \right] ^ {- 1} \left[ \left(Q ^ {T} \mathsf {H} _ {i} Q\right) \otimes x _ {i} \right] \\ = H _ {i} - Q ^ {T} \left(\mathsf {H} _ {i} \otimes x _ {i} ^ {T}\right) (Q \otimes I _ {p}) \left[ \left(Q ^ {T} \otimes I _ {p}\right) \left[ \sum_ {i = 1} ^ {n} \mathsf {H} _ {i} \otimes \left(x _ {i} x _ {i} ^ {T}\right) \right] (Q \otimes I _ {p}) \right] ^ {- 1} \left(Q ^ {T} \otimes I _ {p}\right) (\mathsf {H} _ {i} \otimes x _ {i}) Q \\ = Q ^ {T} \mathsf {H} _ {i} Q - Q ^ {T} \left(\mathsf {H} _ {i} \otimes x _ {i} ^ {T}\right) \left[ \sum_ {i = 1} ^ {n} \left(\mathsf {H} _ {i} \otimes x _ {i} x _ {i} ^ {T}\right) \right] ^ {\dagger} \left(\mathsf {H} _ {i} \otimes x _ {i}\right) Q \\ = Q ^ {T} \vee_ {i} Q, \\ \end{array}
$$
where the penultimate equality is proved as follows.
Let  $A = Q\otimes I_p$  and  $\mathsf{D} = \sum_{i = 1}^{n}\mathsf{H}_i\otimes (x_ix_i^T)$  only in the remaining of this proof. It remains to prove
$$
A \left[ A ^ {T} \mathrm {D} A \right] ^ {- 1} A ^ {T} = \mathrm {D} ^ {\dagger}. \tag {S3.7}
$$
Since  $\mathsf{H}_i\mathbf{1} = 0$ , we have  $\mathsf{D}(\mathbf{1}\otimes I_p) = 0$ . Since  $Q^T\mathbf{1} = 0$  by definition of  $Q$ , we have  $A^T (\mathbf{1}\otimes I_p) = 0$ . If we write the eigen-decomposition of  $\mathsf{D}$  as  $\mathsf{D} = \sum_{i = 1}^{pK}\lambda_iu_iu_i^T$ , then  $u_i^T (\mathbf{1}\otimes I_p) = 0$ . Hence, with  $v_{i} = A^{T}u_{i}$ ,
$$
A ^ {T} \mathrm {D} A = \sum_ {i = 1} ^ {p K} \lambda_ {i} v _ {i} v _ {i} ^ {T}.
$$
Since  $v_{i}^{T}v_{i^{\prime}} = u_{i}^{T}AA^{T}u_{i^{\prime}} = u_{i}^{T}[(I_{K + 1} - \frac{11^{T}}{K + 1})\otimes I_{p}]u_{i^{\prime}} = u_{i}^{T}u_{i^{\prime}} = I(i = i^{\prime})$ , we have
$$
A [ A ^ {T} \mathsf {D} A ] ^ {- 1} A ^ {T} = A \big (\sum_ {i = 1} ^ {p K} \lambda_ {i} ^ {- 1} v _ {i} v _ {i} ^ {T} \big) A ^ {T} = \sum_ {i = 1} ^ {p K} \lambda_ {i} ^ {- 1} u _ {i} u _ {i} ^ {T} = \mathsf {D} ^ {\dagger},
$$
where the second equality uses  $Av_{i} = AA^{T}u_{i} = u_{i}$ . The proof of (S3.7) is complete.
Now we are ready to prove that Theorem 2.1 is a consequence of Theorem S3.1.
Proof of Theorem 2.1. By definition of  $\mathbf{g}_i$  and  $\mathsf{V}_i$ , we have  $\mathbf{1}^T\mathbf{g}_i = 0$  and  $\mathbf{1}^T\mathsf{V}_i = \mathbf{0}^T$ . Thus, we have  $QQ^{T}\mathbf{g}_{i} = \mathbf{g}_{i}$  and  $QQ^{T}\mathsf{V}_{i} = \mathsf{V}_{i}$ . Therefore, we can rewrite the left-hand side of (2.1) (without  $\sqrt{n}\Omega_{jj}^{-1/2}$ ) as
$$
\begin{array}{l} \left(\left(\frac {1}{n} \sum_ {i = 1} ^ {n} \mathbf {g} _ {i} \mathbf {g} _ {i} ^ {T}\right) ^ {1 / 2}\right) ^ {\dagger} \left(\frac {1}{n} \sum_ {i = 1} ^ {n} \mathbf {V} _ {i}\right) \hat {\mathbf {B}} ^ {T} e _ {j} \\ = \left(\left(\frac {1}{n} \sum_ {i = 1} ^ {n} Q Q ^ {T} \mathbf {g} _ {i} \mathbf {g} _ {i} ^ {T} Q Q ^ {T}\right) ^ {1 / 2}\right) ^ {\dagger} \left(\frac {1}{n} \sum_ {i = 1} ^ {n} Q Q ^ {T} \mathbf {V} _ {i} Q Q ^ {T}\right) \hat {\mathbf {B}} ^ {T} e _ {j} \\ = \left(\left(\frac {1}{n} \sum_ {i = 1} ^ {n} Q g _ {i} g _ {i} ^ {T} Q ^ {T}\right) ^ {1 / 2}\right) ^ {\dagger} \left(\frac {1}{n} \sum_ {i = 1} ^ {n} Q V _ {i} Q ^ {T}\right) \hat {\mathbf {B}} ^ {T} e _ {j} \\ = Q \left(\left(\frac {1}{n} \sum_ {i = 1} ^ {n} g _ {i} g _ {i} ^ {T}\right) ^ {1 / 2}\right) ^ {\dagger} Q ^ {T} Q \left(\frac {1}{n} \sum_ {i = 1} ^ {n} V _ {i} Q ^ {T}\right) \hat {\mathsf {B}} ^ {T} e _ {j} \\ = Q \left(\left(\frac {1}{n} \sum_ {i = 1} ^ {n} g _ {i} g _ {i} ^ {T}\right) ^ {1 / 2}\right) ^ {\dagger} \left(\frac {1}{n} \sum_ {i = 1} ^ {n} V _ {i}\right) \hat {B} ^ {T} e _ {j}, \\ \end{array}
$$
where the first equality uses  $QQ^T\mathbf{g}_i = \mathbf{g}_i$  and  $QQ^T\mathsf{V}_i = \mathsf{V}_i$ , the second equality uses  $g_{i} = Q^{T}\mathbf{g}_{i}$  and  $V_{i} = Q^{T}\mathsf{V}_{i}Q$  from Lemma S3.2, the third equality follows from the same argument of (S3.7), and the last equality uses  $Q^T Q = I_K$  and  $\hat{B} = \hat{\mathrm{B}} Q$ .
Therefore, Theorem S3.1 implies that the limiting covariance for the left-hand side of (2.1) is  $QQ^{T} = I_{K} - \frac{11^{T}}{K + 1}$ . This completes the proof.
S4 Proof of Theorem 2.2
We restate Theorem 2.2 for convenience.
$$
\underbrace {\left(I _ {K} + \frac {\mathbf {1} _ {K} \mathbf {1} _ {K} ^ {T}}{\sqrt {K + 1} + 1}\right) R ^ {T}} _ {\text {m a t r i x} \mathbb {R} ^ {K \times (K + 1)}} \underbrace {\sqrt {\frac {n}{\Omega_ {j j}}}} _ {\text {s c a l a r}} \underbrace {\left(\frac {1}{n} \sum_ {i = 1} ^ {n} \mathrm {g} _ {i} \mathrm {g} _ {i} ^ {T}\right) ^ {1 / 2}} _ {\text {m a t r i x} \mathbb {R} ^ {(K + 1) \times (K + 1)}} ^ {\dagger} \underbrace {\left(\frac {1}{n} \sum_ {i = 1} ^ {n} \mathrm {V} _ {i} R\right)} _ {\mathbb {R} ^ {(K + 1) \times K}} \underbrace {\hat {A} ^ {T} e _ {j}} _ {\mathbb {R} ^ {K}} \xrightarrow {\mathrm {d}} N (\mathbf {0} _ {K}, I _ {K}) \tag {2.2}
$$
where  $\mathbf{g}_i$  is defined in (1.7) and  $\mathsf{V}_i$  is defined in Theorem 2.1. Furthermore, for the same  $j\in [p]$
The proof is a direct consequence of Theorem 2.1.
Proof of Theorem 2.2. By definition of  $\hat{A}$  in (1.9), we have  $\hat{A} = \hat{\mathbf{B}}(I_K, -\mathbf{1}_K)^T$  and
$$
\hat {A} \left(I _ {K}, \mathbf {0} _ {K}\right) = \hat {\mathrm {B}} \left(I _ {K}, - \mathbf {1} _ {K}\right) ^ {T} \left(I _ {K}, \mathbf {0} _ {K}\right) = \hat {\mathrm {B}} \left(I _ {K + 1} - e _ {K + 1} \mathbf {1} ^ {T}\right) = \hat {\mathrm {B}} - (\hat {\mathrm {B}} e _ {K + 1}) \mathbf {1} ^ {T},
$$
which is of the form  $\hat{\mathbf{B}} - b\mathbf{1}^T$  with  $b = \hat{\mathsf{B}} e_{K + 1}$ . Therefore,  $\hat{A}\left(I_K,\mathbf{0}_K\right)$  is also a solution of (1.5). Taking  $\hat{\mathbf{B}}$  in Theorem 2.1 to be  $\hat{A}\left(I_K,\mathbf{0}_K\right) = \hat{A} R^T$  gives the desired  $\chi^2$  result (2.3) and
$$
\sqrt {n} \Omega_ {j j} ^ {- 1 / 2} \left(\left(\frac {1}{n} \sum_ {i = 1} ^ {n} \mathbf {g} _ {i} \mathbf {g} _ {i} ^ {T}\right) ^ {1 / 2}\right) ^ {\dagger} \left(\frac {1}{n} \sum_ {i = 1} ^ {n} \mathbf {V} _ {i}\right) R \hat {A} ^ {T} e _ {j} \xrightarrow {\mathrm {d}} N \left(0, I _ {K + 1} - \frac {\mathbf {1 1} ^ {T}}{K + 1}\right). \tag {S4.1}
$$
Multiplying  $(R^T (I_{K + 1} - \frac{\mathbf{11}^T}{K + 1})R)^{-1 / 2}R^T$  to the left of the above display gives the desired normality result (2.2) by observing  $(R^{T}(I_{K + 1} - \frac{\mathbf{11}^{T}}{K + 1})R)^{-1 / 2} = (I_{K} + \frac{\mathbf{1}_{K}\mathbf{1}_{K}^{T}}{\sqrt{K + 1} + 1})$ . This completes the proof.
S5 Preliminary results for proving Theorem S3.1
S5.1 Results for general loss functions
In this subsection, we will work under the following assumptions with a general convex loss function. Later in Section S6, we will apply the general results of this subsection to the multinomial logistic loss discussed in Section S3.1.
Assumption S5.1. Suppose we have data  $(Y,X)$ , where  $Y\in \mathbb{R}^{n\times (K + 1)}$  with rows  $(y_{1},\dots,y_{n})$ , and  $X\in \mathbb{R}^{n\times p}$  has i.i.d. rows  $(x_{1},\ldots ,x_{n})$  with  $x_{i}\sim N(\mathbf{0},\Sigma)$  and invertible  $\boldsymbol{\Sigma}$ . The observations  $(y_{i},x_{i})_{i\in [n]}$  are i.i.d. and  $y_{i}$  has the form  $y_{i} = f(U_{i},x_{i}^{T}B^{*})$  for some deterministic function  $f$ , deterministic  $B^{*}\in \mathbb{R}^{p\times K}$ , and latent random variable  $U_{i}$  independent of  $x_{i}$ . Assume  $p / n\leq \delta^{-1} < 1$ .
Assumption S5.2. Given data  $(Y,X)$ , consider twice continuously differentiable and strictly convex loss functions  $(L_{i})_{i\in [n]}$  with each  $L_{i}:\mathbb{R}^{K}\to \mathbb{R}$  depending on  $y_{i}$  but not on  $x_{i}$ .
Provided that the following minimization problem admits a solution, define
$$
\hat {B} (Y, X) = \operatorname * {a r g   m i n} _ {B \in \mathbb {R} ^ {p \times K}} \sum_ {i = 1} ^ {n} L _ {i} \left(B ^ {T} x _ {i}\right).
$$