SlowGuess/NeurIPS_2023_OCR huggingface dataset:
If  $(Y,X),(Y,\tilde{X})\in U$  , we have  $\| X\hat{B} Q^T\| _F^2 +\| \tilde{X}\tilde{B} Q^T\| _F^2\leq 2n\tau$  . That is,
$$
\sum_ {i = 1} ^ {n} \left(\| Q \hat {B} ^ {T} x _ {i} \| ^ {2} + \| Q \tilde {B} ^ {T} \tilde {x} _ {i} \| ^ {2}\right) \leq 2 n \tau .
$$
Define the index set
$$
I = \left\{i \in [ n ]: \| Q \hat {B} ^ {T} x _ {i} \| ^ {2} + \| Q \tilde {B} ^ {T} \tilde {x} _ {i} \| ^ {2} \leq \frac {2 \tau}{\alpha} \right\},
$$
then we have  $|I| \geq (1 - \alpha)n$  by Markov's inequality. Thus, for all  $i \in I$ , we have  $\| Q\hat{B}^T x_i\|_\infty \vee \| Q\tilde{B}^T\tilde{x}_i\|_\infty \leq \sqrt{\frac{2\tau}{\alpha}}$ .
Applying Lemma S6.2 with  $r = \sqrt{\frac{2\tau}{\alpha}}$  gives
$$
\begin{array}{l} \min  \_ \nabla^ {2} L _ {i} (u) \succeq c _ {*} I _ {K}, \\ \| Q u \| _ {\infty} \leq \sqrt {\frac {2 \tau}{\alpha}} \\ \end{array}
$$
where  $c_{*}$  is a constant depending on  $(K,\tau ,\alpha)$ . Therefore,
$$
\begin{array}{l} c _ {*} \| P _ {I} (X \hat {B} - \tilde {X} \tilde {B}) \| _ {F} ^ {2} = c _ {*} \sum_ {i \in I} \| \hat {B} ^ {T} x _ {i} - \tilde {B} ^ {T} \tilde {x} _ {i} \| ^ {2} \\ \leq \sum_ {i \in I} \langle \hat {B} ^ {T} x _ {i} - \tilde {B} ^ {T} \tilde {x} _ {i}, \nabla L _ {i} (\hat {B} ^ {T} x _ {i}) - \nabla L _ {i} (\tilde {B} ^ {T} \tilde {x} _ {i}) \rangle \\ \leq \sum_ {i = 1} ^ {n} \langle \hat {B} ^ {T} x _ {i} - \tilde {B} ^ {T} \tilde {x} _ {i}, \nabla L _ {i} (\hat {B} ^ {T} x _ {i}) - \nabla L _ {i} (\tilde {B} ^ {T} \tilde {x} _ {i}) \rangle \\ = \langle X \hat {B} - \tilde {X} \tilde {B}, G - \tilde {G} \rangle \\ = - \langle (X - \tilde {X}) (\hat {B} - \tilde {B}), G \rangle + \langle (X - \tilde {X}) \hat {B}, G - \hat {G} \rangle . \\ \end{array}
$$
We next bound the first line from below by expanding the squares,
$$
\begin{array}{l} \| P _ {I} (X \hat {B} - \tilde {X} \tilde {B}) \| _ {F} ^ {2} = \| P _ {I} \tilde {X} (\hat {B} - \tilde {B}) + P _ {I} (X - \tilde {X}) \hat {B} \| _ {F} ^ {2} \\ \geq \| P _ {I} \tilde {X} (\hat {B} - \tilde {B}) \| _ {F} ^ {2} + 2 \langle P _ {I} \tilde {X} (\hat {B} - \tilde {B}), P _ {I} (X - \tilde {X}) \hat {B} \rangle \\ \geq n \phi_ {*} ^ {2} \| \Sigma^ {1 / 2} (\hat {B} - \tilde {B}) \| _ {F} ^ {2} + 2 \langle \tilde {X} (\hat {B} - \tilde {B}), P _ {I} (X - \tilde {X}) \hat {B} \rangle , \\ \end{array}
$$
where in the last inequality we use the constant  $\phi^{*}$  in (S6.3). Therefore, we obtain
$$
\begin{array}{l} c _ {*} \phi_ {*} ^ {2} n \| \Sigma^ {1 / 2} (\hat {B} - \tilde {B}) \| _ {F} ^ {2} \\ \leq - \langle (X - \tilde {X}) (\hat {B} - \tilde {B}), G \rangle + \langle (X - \tilde {X}) \hat {B}, G - \tilde {G} \rangle - 2 c _ {*} \langle \tilde {X} (\hat {B} - \tilde {B}), P _ {I} (X - \tilde {X}) \hat {B} \rangle . \\ \end{array}
$$
Together with the inequality that  $\| G - \tilde{G}\| _F^2\leq \langle X\hat{B} -\tilde{X}\tilde{B},G - \tilde{G}\rangle$  , we obtain
$$
\begin{array}{l} c _ {*} \phi_ {*} ^ {2} n \| \Sigma^ {1 / 2} (\hat {B} - \tilde {B}) \| _ {F} ^ {2} + \| G - \tilde {G} \| _ {F} ^ {2} \\ \leq - 2 \langle (X - \tilde {X}) (\hat {B} - \tilde {B}), G \rangle + 2 \langle (X - \tilde {X}) \hat {B}, G - \tilde {G} \rangle - 2 c _ {*} \langle \tilde {X} (\hat {B} - \tilde {B}), P _ {I} (X - \tilde {X}) \hat {B} \rangle \\ \leq (4 + 2 c _ {*} \phi^ {*}) \| (X - \tilde {X}) \Sigma^ {- 1 / 2} \| _ {o p} \big (\| \Sigma^ {1 / 2} (\hat {B} - \tilde {B}) \| _ {F} \vee \frac {\| G - \tilde {G} \| _ {F}}{\sqrt {n}} \big) \big (\| \Sigma^ {1 / 2} \hat {B} \| _ {F} \vee \frac {\| G \| _ {o p}}{\sqrt {n}} \big) \sqrt {n}, \\ \end{array}
$$
where we bound  $\langle \tilde{X} (\hat{B} -\tilde{B}),P_I(X - \tilde{X})\hat{B}\rangle$  by definition of  $\phi^{*}$
$$
\begin{array}{l} \langle \tilde {X} (\hat {B} - \tilde {B}), P _ {I} (X - \tilde {X}) \hat {B} \rangle \\ = \left\langle \Sigma^ {1 / 2} (\hat {B} - \tilde {B}), \Sigma^ {- 1 / 2} \tilde {X} ^ {T} P _ {I} (X - \tilde {X}) \hat {B} \right\rangle \\ \leq \| \Sigma^ {1 / 2} (\hat {B} - \tilde {B}) \| _ {F} \| P _ {I} \tilde {X} \Sigma^ {- 1 / 2} \| _ {o p} \| (X - \tilde {X}) \Sigma^ {- 1 / 2} \| _ {o p} \| \Sigma^ {1 / 2} \hat {B} \| _ {F} \\ \leq \sqrt {n} \phi^ {*} \| \Sigma^ {1 / 2} (\hat {B} - \tilde {B}) \| _ {F} \| (X - \tilde {X}) \Sigma^ {- 1 / 2} \| _ {o p} \| \Sigma^ {1 / 2} \hat {B} \| _ {F}. \\ \end{array}
$$
Now we derive a bound of the form  $\| \Sigma^{1 / 2}\hat{B}\| _F\lesssim \| G\| _F / \sqrt{n}$ . To this end, since  $\phi_{*}\| \Sigma^{1 / 2}\hat{B}\|_{F}\leq$ $\| P_I X\hat{B}\| _F / \sqrt{n}\leq \| X\hat{B}\| _F / \sqrt{n} = \| X\hat{B} Q^T\| _F / \sqrt{n}\leq \sqrt{\tau}$
Applying Lemma S6.1 to  $M = X\hat{B}$ , we have  $\frac{1}{n}\sum_{i=1}^{n}g_{i}g_{i}^{T} \succeq m_{*}I_{K}$ . Therefore,
$$
\frac {1}{n} \| G \| _ {F} ^ {2} = \frac {1}{n} \sum_ {i = 1} ^ {n} \| g _ {i} \| ^ {2} = \frac {1}{n} \sum_ {i = 1} ^ {n} \mathrm {T r} (g _ {i} g _ {i} ^ {T}) \geq K m _ {*}.
$$
$$
\phi_ {*} ^ {2} \| \Sigma^ {1 / 2} \hat {B} \| _ {F} ^ {2} \leq \tau \leq \frac {\tau}{K m _ {*} (I)} \| G \| _ {F} ^ {2} / n.
$$
In conclusion, if  $\{(Y,X),(Y,\tilde{X})\} \subset U^{*}$  then
$$
\sqrt {n} \| \Sigma^ {1 / 2} (\hat {B} - \tilde {B}) \| _ {F} + \| G - \tilde {G} \| _ {F} \leq C n ^ {- 1 / 2} \| (X - \tilde {X}) \Sigma^ {- 1 / 2} \| _ {o p} \| G \| _ {F} \leq C K \| (X - \tilde {X}) \Sigma^ {- 1 / 2} \| _ {o p},
$$
where  $C$  is a constant depending on  $(K,\gamma ,\tau ,\alpha)$  only. Note that  $\| G\| _F\leq \sqrt{nK}$  since all entries of  $G$  are in  $[-1,1]$ .
Lemma S6.4. If  $p / n \leq \delta^{-1} < (1 - \alpha)$  for some  $\alpha \in (0,1)$  and  $\delta > 1$ . If  $(Y,X) \in U$  and  $X \in U_x$ , where  $U_x$  is defined in Lemma S6.3, we have
$$
\frac {1}{n} \sum_ {i = 1} ^ {n} H _ {i} \otimes \left(x _ {i} x _ {i} ^ {T}\right) \succeq c _ {1} \left(I _ {K} \otimes \Sigma\right),
$$
where  $c_{1}$  is a positive constant depending on  $(K,\tau ,\alpha ,\phi_{*})$  only.
Proof of Lemma S6.4. If  $(Y,X)\in U$ , we have  $\| X\hat{B} Q^T\| _F^2\leq n\tau$ . Define the index set
$$
I = \{i \in [ n ]: \| Q \hat {B} ^ {T} x _ {i} \| \leq \frac {\tau}{\alpha} \},
$$
then we have  $|I| \geq (1 - \alpha)n$  by Markov's inequality. Therefore, for any  $i \in I$ ,  $\|Q\hat{B}^T x_i\|_{\infty} \leq \frac{\tau}{\alpha}$ . Applying Lemma S6.2 with  $u = \hat{B}^T x_i$  and  $r = \frac{\tau}{\alpha}$ , we have for any  $i \in I$ ,  $H_i = \nabla^2 L_i(\hat{B}^T x_i) \succeq c_* I_K$  for some positive constant  $c_*$  depending on  $(K, \tau, \alpha)$  only. Therefore, if  $(Y, X) \in U$  and  $X \in U_x$ , we have
$$
\begin{array}{l} \frac {1}{n} \sum_ {i = 1} ^ {n} H _ {i} \otimes \left(x _ {i} x _ {i} ^ {T}\right) \succeq \frac {1}{n} \sum_ {i \in I} H _ {i} \otimes \left(x _ {i} x _ {i} ^ {T}\right) \\ \succeq c _ {*} \frac {1}{n} \sum_ {i \in I} I _ {K} \otimes (x _ {i} x _ {i} ^ {T}) \\ = c _ {*} \left(I _ {K} \otimes \frac {X ^ {T} P _ {I} X}{n}\right) \\ \succeq c _ {*} \phi_ {*} (I _ {K} \otimes \Sigma) \\ = c _ {1} \left(I _ {K} \otimes \Sigma\right), \\ \end{array}
$$
where  $P_{I} = \sum_{i\in I}e_{i}e_{i}^{T}$  and  $c_{1}$  is a positive constant depending on  $(K,\tau ,\alpha ,\phi_{*})$  only.
S6.3 Proof of Theorem S3.1
The proof of Theorem S3.1 is a direct consequence of Theorem S5.1 by noting
$$
\sqrt {n} \Omega_ {j j} ^ {- 1 / 2} \left(\frac {1}{n} \sum_ {i = 1} ^ {n} g _ {i} g _ {i} ^ {T}\right) ^ {- 1 / 2} \left(\frac {1}{n} \sum_ {i = 1} ^ {n} V _ {i}\right) \hat {B} ^ {T} e _ {j} = \Omega_ {j j} ^ {- 1 / 2} (G ^ {T} G) ^ {- 1 / 2} V \hat {B} ^ {T} e _ {j},
$$
which is a consequence of the identities  $G = \sum_{i=1}^{n} e_i g_i^T$  and  $V = \sum_{i=1}^{n} V_i$ .
It thus remains to verify the conditions (1) and (2) in Theorem S5.1 from the assumptions in Theorem S3.1.
Applying Lemma S6.3 with  $\alpha$  chosen as  $1 - \delta^{-1/2}$ , we have for  $\{(Y, X), (Y, \tilde{X})\} \subset U^{*}$
$$
\left\| G (Y, X) - G (Y, \tilde {X}) \right\| _ {F} \leq L \| (X - \tilde {X}) \Sigma^ {- 1 / 2} \| _ {F},
$$
where  $L$  is a positive constant depending on  $(K,\gamma ,\tau ,\delta)$  only.
Apply Lemma S6.4 with the same  $\alpha = 1 - \delta^{-1 / 2}$ , we have for  $(Y,X)\in U^{*}$
$$
\frac {1}{n} \sum_ {i = 1} ^ {n} H _ {i} \otimes \left(x _ {i} x _ {i} ^ {T}\right) \succeq c _ {*} \left(I _ {K} \otimes \Sigma\right),
$$
where  $c_{*}$  is a positive constant depending on  $(K,\tau ,\delta)$  only.
Applying Lemma S6.1 with  $M = X\hat{B}$ , we have for  $(Y,X)\in U^{*}$
where  $m_*$  is a positive constant depending on  $(K, \gamma, \tau)$  only. Therefore, the conditions (1) and (2) in Theorem S5.1 hold when the multinomial logistic loss is used. This completes the proof of Theorem S3.1.
S7 Other proof
S7.1 Proof of Equation (1.12) (Classical asymptotic theory with fixed  $p$ )
Here we provide a derivation of the asymptotic distribution of MLE under classical setting, where  $p$  is fixed and  $n$  tends to infinity.
We first calculate the Fisher information matrix of the multinomial logistic log-odds model (1.8) with covariate  $x \sim N(\mathbf{0}, \Sigma)$  and response  $\mathsf{y} \in \{0, 1\}^{K+1}$  one-hot encoded satisfying  $\sum_{k=1}^{K+1} \mathsf{y}_k = 1$ . Note that the model (1.8) can be rewritten as
$$
\mathbb {P} (\mathsf {y} _ {k} = 1 | x) = \frac {\exp (x ^ {T} A ^ {*} e _ {k})}{1 + \sum_ {k ^ {\prime} = 1} ^ {K} \exp (x ^ {T} A ^ {*} e _ {k ^ {\prime}})}, \quad \forall k \in \{1, \dots , K \}
$$
$$
\mathbb {P} (\mathsf {y} _ {K + 1} = 1 | x) = \frac {1}{1 + \sum_ {k ^ {\prime} = 1} ^ {K} \exp (x ^ {T} A ^ {*} e _ {k ^ {\prime}})}.
$$
The likelihood function of a parameter  $A\in \mathbb{R}^{p\times K}$  is
$$
\begin{array}{l} L (A) = \prod_ {k = 1} ^ {K} \left[ \frac {\exp \left(x ^ {T} A e _ {k}\right)}{1 + \sum_ {k ^ {\prime} = 1} ^ {K} \exp \left(x ^ {T} A e _ {k ^ {\prime}}\right)} \right] ^ {\mathcal {Y} _ {k}} \left[ \frac {1}{1 + \sum_ {k ^ {\prime} = 1} ^ {K} \exp \left(x ^ {T} A e _ {k ^ {\prime}}\right)} \right] ^ {\mathcal {Y} _ {K + 1}} \\ = \prod_ {k = 1} ^ {K} \left[ \exp \left(x ^ {T} A e _ {k}\right) \right] ^ {\mathcal {Y} _ {k}} \frac {1}{1 + \sum_ {k ^ {\prime} = 1} ^ {K} \exp \left(x ^ {T} A e _ {k ^ {\prime}}\right)}, \\ \end{array}
$$
where we used  $\sum_{k=1}^{K+1} y_k = 1$ . Thus, the log-likelihood function is
$$
\ell (A) = \sum_ {k = 1} ^ {K} \mathcal {y} _ {k} x ^ {T} A e _ {k} - \log \left[ 1 + \sum_ {k ^ {\prime} = 1} ^ {K} \exp \left(x ^ {T} A e _ {k ^ {\prime}}\right) \right].
$$
It is more convenient to calculate the Fisher information matrix on the vector space  $\mathbb{R}^{pK}$  instead of the matrix space  $\mathbb{R}^{p\times K}$ . To this end, let  $\theta = \operatorname{vec}(A^T)$ , then  $x^T A e_k = e_k^T A^T x = (x^T\otimes e_k^T)\operatorname{vec}(A^T) = (x^T\otimes e_k^T)\theta$ , and the log-likelihood function parameterized by  $\theta$  is
$$
\ell (\theta) = \sum_ {k = 1} ^ {K} y _ {k} (x ^ {T} \otimes e _ {k} ^ {T}) \theta - \log [ 1 + \sum_ {k ^ {\prime} = 1} ^ {K} \exp ((x ^ {T} \otimes e _ {k ^ {\prime}} ^ {T}) \theta) ].
$$
By multivariate calculus, we obtain the Fisher information matrix evaluated at  $\theta^{*} = \mathrm{vec}(A^{*T})$ ,
$$
\begin{array}{l} \mathcal {I} \left(\theta^ {*}\right) = - \mathbb {E} \left[ \frac {\partial}{\partial \theta} \frac {\partial \ell (\theta)}{\partial \theta^ {T}} \right] \Big | _ {\theta = \theta^ {*}} \\ = \mathbb {E} \left[ \left(x x ^ {T}\right) \otimes \left(\operatorname {d i a g} \left(\pi^ {*}\right) - \pi^ {*} \pi^ {* T}\right) \right], \\ \end{array}
$$
where  $\pi^{*}\in \mathbb{R}^{K}$  with  $k$ -th entry  $\pi_k^* = \frac{\exp(x^TA^*e_k)}{1 + \sum_{k' = 1}^{K}\exp(x^TA^*e_{k'})}$  for each  $k\in [K]$ .
From classical maximum likelihood theory, for instance [Van der Vaart, 1998, Chapter 5], we have
$$
\sqrt {n} (\hat {\theta} - \theta^ {*}) \xrightarrow {\mathrm {d}} N (\mathbf {0}, \mathcal {I} _ {\theta^ {*}} ^ {- 1}),
$$
where  $\hat{\theta} = \operatorname{vec}(\hat{A}^T)$  and  $\hat{A}$  is the MLE of  $A^{*}$ . Furthermore, if the  $j$ -th covariate is independent of the response, we know  $e_j^T A^* = \mathbf{0}^T$ , then
$$
\sqrt {n} \hat {A} ^ {T} e _ {j} = \sqrt {n} (\hat {A} ^ {T} e _ {j} - A ^ {* T} e _ {j}) = \sqrt {n} (e _ {j} ^ {T} \otimes I _ {K}) (\hat {\theta} - \theta^ {*}) \xrightarrow {\mathrm {d}} N (\mathbf {0}, S _ {j}),
$$
$$
\begin{array}{l} S _ {j} = \left(e _ {j} ^ {T} \otimes I _ {K}\right) \mathcal {I} _ {\theta^ {*}} ^ {- 1} \left(e _ {j} \otimes I _ {K}\right) \\ = e _ {j} ^ {T} \operatorname {c o v} (x) ^ {- 1} e _ {j} \left[ \mathbb {E} \left(\operatorname {d i a g} \left(\pi^ {*}\right) - \pi^ {*} \pi^ {* T}\right) \right] ^ {- 1} \\ \end{array}
$$
holds by the independence between the  $j$ -th covariate and the response under  $H_0$  in (1.10). This completes the proof of Equation (1.12).