SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Note that  $\mathsf{CN}_{\alpha,\beta}$  normalizes the RHS of (18) into a standard Gaussian. Indeed, define
$$
Z ^ {(\beta)} \triangleq \frac {1}{\mathbb {C N} _ {\alpha , \beta}} \sum_ {j \notin \{\alpha , \beta \}} \lambda_ {j} \widehat {\boldsymbol {h}} _ {\beta , \alpha} [ j ] \boldsymbol {x} _ {\text {t e s t}} [ j ] \sim N (0, 1). \tag {20}
$$
Since  $\alpha, \beta \in [k]$  are favored, we have  $\lambda_{\alpha} = \lambda_{\beta} = \lambda_{F}$ . Hence, an equivalent condition for misclassification is that there exists some  $\beta \neq \alpha$ ,  $\beta \in [k]$  such that
$$
\frac {\lambda_ {F}}{\mathbb {C N} _ {\alpha , \beta}} \left(\widehat {\boldsymbol {h}} _ {\alpha , \beta} [ \alpha ] \boldsymbol {x} _ {\text {t e s t}} [ \alpha ] - \widehat {\boldsymbol {h}} _ {\beta , \alpha} [ \beta ] \boldsymbol {x} _ {\text {t e s t}} [ \beta ]\right) <   Z ^ {(\beta)}. \tag {21}
$$
We now translate the above criterion into sufficient conditions for correct classification and misclassification and analyze these two cases separately.
Correct classification: For correct classification, it suffices for the minimum value of the LHS of Eq. (21) to outcompete the maximum value of the RHS, where the max is taken over  $\beta \in [k],\beta \neq \alpha$  Some algebra, as in Subramanian et al. (2022), shows that we correctly classify if
$$
\underbrace {\frac {\operatorname* {m i n} _ {\beta} \lambda_ {F} \widehat {\boldsymbol {h}} _ {\alpha , \beta} [ \alpha ]}{\operatorname* {m a x} _ {\beta} \mathsf {C N} _ {\alpha , \beta}}} _ {\text {S U / C N r a t i o}} \left(\underbrace {\min  _ {\beta} \left(\boldsymbol {x} _ {\text {t e s t}} [ \alpha ] - \boldsymbol {x} _ {\text {t e s t}} [ \beta ]\right)} _ {\text {c l o s e s t f e a t u r e m a r g i n}} - \underbrace {\max  _ {\beta} | \boldsymbol {x} _ {\text {t e s t}} [ \beta ] |} _ {\text {l a r g e s t c o m p e t e n g f e a t u r e}} \cdot \underbrace {\max  _ {\beta} \left| \frac {\widehat {\boldsymbol {h}} _ {\alpha , \beta} [ \alpha ] - \widehat {\boldsymbol {h}} _ {\beta , \alpha} [ \beta ]}{\widehat {\boldsymbol {h}} _ {\alpha , \beta} [ \alpha ]} \right|} _ {\text {s u r v i v a l v a r i a t i o n}}} _ {> \underbrace {\max  _ {\beta} Z ^ {(\beta)}} _ {\text {n o r m a l i z e d c o n t a m i n a t i o n}}}. \right. \tag {22}
$$
We will show that under the conditions specified in Conjecture 3.1, with high probability, the relevant survival to contamination ratio  $\mathsf{SU} / \mathsf{CN}$  grows at a polynomial rate  $n^v$  for some  $v > 0$ , whereas the term in the parentheses shrinks at a subpolynomial rate  $\omega (n^{-\delta})$  for any  $\delta >0$ . Further, by standard subgaussian maximal inequalities, the magnitudes of the normalized contamination is no more than  $O(\sqrt{\log(nk)})$  with high probability. Thus, with high probability the LHS outcompetes the RHS, leading to correct classification. See Section 4.1 for more discussion on how we prove tight bounds on the survival-to-contamination ratios.
Misclassification: On the other hand, for misclassification it suffices for the maximum absolute value of the LHS of Eq. (21) to be outcompeted by the maximum value of the RHS. Some manipulations yield the following sufficient condition for misclassification:
$$
\underbrace {\frac {\operatorname* {m a x} _ {\beta} \lambda_ {F} \left(\left| \widehat {\boldsymbol {h}} _ {\alpha , \beta} [ \alpha ] \right| + \left| \widehat {\boldsymbol {h}} _ {\beta , \alpha} [ \beta ] \right|\right)}{\operatorname* {m i n} _ {\beta} \mathsf {C N} _ {\alpha , \beta}}} _ {\text {S U / C N r a t i o}} \cdot \underbrace {\underset {\gamma \in [ k ]} {\max } | \boldsymbol {x} _ {\text {t e s t}} [ \gamma ] |} _ {\text {l a r g e s t l a b e l - d e f i n i n g f e a t u r e}} <   \underbrace {\underset {\beta} {\max } Z ^ {(\beta)}} _ {\text {n o r m a l i z e d c o n t a m i n a t i o n}}. \tag {23}
$$
We show that within the misclassification regimes in Conjecture 3.1, the survival-to-contamination ratio  $\mathsf{SU} / \mathsf{CN}$  shrinks at a polynomial rate  $n^{-w}$  for some  $w > 0$ . By standard subgaussian maximal inequalities, the largest label-defining feature is  $O(\sqrt{\log(nk)})$  with high probability. Gaussian anticoncentration implies that for some  $\beta \neq \alpha, \beta \in [k], Z^{(\beta)}$  outcompetes the LHS with probability at least  $\frac{1}{2} - o(1)$ . Hence, we conclude that the model will misclassify with rate at least  $\frac{1}{2}$  asymptotically.
Let us now describe how to boost the misclassification rate to  $1 - o(1)$ . Notice that the above argument only considered the competition between the LHS of Eq. (23) and one of the  $Z^{(\beta)}$ 's on the RHS instead of the maximum  $Z^{(\beta)}$ . It's not hard to see from the definition of  $Z^{(\beta)}$  in Eq. (20) that the  $Z^{(\beta)}$  are jointly Gaussian. For intuition's sake, assuming the  $Z^{(\beta)}$  were independent, then  $\max_{\beta} Z^{(\beta)}$  would outcompete with probability  $(\frac{1}{2} - o(1))^{k-1}$ .
In reality, the  $Z^{(\beta)}$  are correlated, but we are able to show that the maximum correlation between the  $Z^{(\beta)}$  is  $\frac{1}{2} + o(1)$  with high probability. An application of Slepian's lemma (Slepian (1962)) and some explicit bounds on orthant probabilities (Pinasco et al. (2021)) implies that  $\max_{\beta} Z^{(\beta)} > 0$  with probability at least  $1 - \frac{1}{k^{1 + o(1)}}$ . Another application of anticoncentration implies that  $\max_{\beta} Z^{(\beta)} > n^{-w}$  with probability  $1 - o(1)$ , which finishes off the proof.
4.1 Bounding the survival-to-contamination ratio
Note that the critical survival-to-contamination ratio appears in both Eqs. (22) and (23). The most involved part of the proof is nailing down the correct order of growth of the survival to contamination ratio; a similar analysis tightly bounds the survival variation and the correlation structure of the  $Z^{(\beta)}$ .
To understand the relative survival and contamination, we must analyze the bilinear forms  $\widehat{h}_{\alpha,\beta}[j] = z_j^\top A^{-1}\Delta y$ . Similarly, to control the correlation of the  $Z^{(\beta)}$ , we must understand the correlation between the  $\widehat{h}_{\alpha,\beta}$  vectors, which reduces to understanding the bilinear forms  $z_j^\top A^{-1}y_\alpha$  for  $j \in [d], \alpha \in [k]$ . The main source of inspiration for bounding these bilinear forms is the heuristic style of calculation carried out in Appendix K of Subramanian et al. (2022) that leads to Conjecture 3.1.
To simplify the discussion, we temporarily restrict to the regime where regression fails  $(q + r > 1)$ . However, our main technical tool seamlessly generalizes to the regime where regression works  $(q + r < 1)$ . In the regime where regression fails,  $A^{-1}$  turns out to have a flat spectrum:  $A^{-1} \approx \alpha I$  for some constant  $\alpha > 0$ . Assume for now that  $A^{-1}$  is exactly equal to a scaled identity matrix. Then the survival is proportional to  $z_{\alpha}^{\top} \Delta y$ , which is a random inner product. Similarly, to bound the contamination terms we must control the random inner product  $z_{j}^{\top} \Delta y$  for  $j \notin \{\alpha, \beta\}$ .
Since  $\Delta y$  is a sparse vector — it only has  $\frac{2n}{k}$  nonzero entries in expectation — a quick computation reveals that  $\mathbb{E}[z_{\alpha}^{\top}\Delta y] = \tilde{O}\left(\frac{n}{k}\right)$  and  $\mathbb{E}[z_j^\top \Delta y] = 0$ . The deciding factor, then, is how tightly these quantities concentrate around their means. A naive application of Hoeffding implies a concentration radius of order  $\tilde{O} (\sqrt{n})$ , which would lead to looseness in the overall result. The hope is to exploit sparsity to get a concentration radius of order  $\tilde{O} (\sqrt{n / k})$ . This is where our new technical tool Theorem 4.1 comes in, which may be of independent interest; we present it in the following section.
4.2 A new variant of the Hanson-Wright inequality
In reality, even in the regime where regression fails,  $A^{-1}$  is not actually perfectly flat. Even worse, in the regime where regression works,  $A^{-1}$  is actually spiked. Thus, we cannot simply reduce the bilinear form  $z_{j}^{\top}A^{-1}\Delta y$  to an inner product. Instead, we turn to the well-known Hanson-Wright inequality (Rudelson and Vershynin, 2013), which tells us that quadratic forms of random vectors with independent, mean zero, subgaussian entries concentrate around their mean. It was used extensively to study binary classification (Muthukumar et al., 2021), and multiclass classification (Subramanian et al., 2022; Wu and Sahai, 2023).
However, just as Hoeffding is loose, so too is the standard form of Hanson-Wright, because it also does not exploit sparsity. This motivates a new variant of Hanson-Wright which fully leverages the (soft) sparsity inherent to multiclass problems with an increasing number of classes. We now formally define the notions of soft and hard sparsity.
Definition 2 (Soft and hard sparsity). For  $\pi \leq 1$ , we say that random vector  $\mathbf{y} = (Y_i)_{i=1}^n$  has soft sparsity at level  $\pi$  if  $|Y_i| \leq 1$  almost surely and  $\operatorname{Var}(Y_i) \leq \pi$  for all  $i$ . On the other hand, we say that  $\mathbf{y}$  has hard sparsity at level  $\pi$  if at most a  $\pi$  fraction of the  $Y_i$  are nonzero.
In particular, our variant Theorem 4.1 below requires that one of the vectors in the bilinear form has soft sparsity at level  $\pi$ . Throughout, one should think of  $\pi = o(1)$ , and for us indeed  $\pi = O\left(\frac{1}{k}\right)$ . One can check that a bounded random vector  $\mathbf{y}$  with hard sparsity level  $\pi$  must also have soft sparsity at level  $O(\pi)$ , so soft sparsity is more general for bounded random vectors. In Table 1 we compare our variant with several variants of Hanson-Wright which have appeared in the literature, some of which involve hard sparsity.
Define the subgaussian norm  $\| \xi \|_{\psi_2}$  (Vershynin, 2018) as
$$
\left\| \xi \right\| _ {\psi_ {2}} = \inf  _ {K > 0} \left\{K: \mathbb {E} \exp \left(\xi^ {2} / K ^ {2}\right) \leq 2 \right\}, \tag {24}
$$
Theorem 4.1 (Hanson-Wright for bilinear forms with soft sparsity). Let  $\boldsymbol{x} = (X_1, \ldots, X_n) \in \mathbb{R}^n$  and  $\boldsymbol{y} \in (Y_1, \ldots, Y_n) \in \mathbb{R}^n$  be random vectors such that  $(X_i, Y_i)$  are independent pairs of (possibly correlated) centered random variables such that  $\| X_i \|_{\psi_2} \leq K$  and  $Y_i$  has soft sparsity at level  $\pi$ , i.e.  $|Y_i| \leq 1$  almost surely, and  $\mathbb{E}[Y_i^2] \leq \pi$ . Assume that conditioned on  $Y_j$ ,  $\| X_j \|_{\psi_2} \leq K$ . Then there exists an absolute constant  $c > 0$  such that for all  $M \in \mathbb{R}^{n \times n}$  and  $\epsilon \geq 0$  we have
$$
\Pr \left[ | \boldsymbol {x} ^ {\top} \boldsymbol {M} \boldsymbol {y} - \mathbb {E} [ \boldsymbol {x} ^ {\top} \boldsymbol {M} \boldsymbol {y} ] | > \epsilon \right] \leq 2 \exp \left(- c \min  \left\{\frac {\epsilon^ {2}}{K ^ {2} \pi \| \boldsymbol {M} \| _ {F} ^ {2}}, \frac {\epsilon}{K \| \boldsymbol {M} \| _ {2}} \right\}\right). \tag {25}
$$
The full proof of Theorem 4.1 is deferred to Appendix G. The main proof techniques are heavily inspired by those of Rudelson and Vershynin (2013); Zhou (2019); Park et al. (2022). However, the
Variant	Assumptions on y	Concentration radius
Classic quadratica: xT Mx	same as x	O(||M||F)
Sparse bilinearb: xT M(γ○y)	γi ~ Ber(π), independ. of Xi but not Yi	O(√π||M||F)
Sparse bilinearc: xT M(γ○y)	γi ~ Ber(π), independ. of Yi but not Xi	O(√π||M||F)
Theorem 4.1: xT My	|Yi| ≤ 1 a.s., EYi2 ≤ π	O(√π||M||F)
Table 1: Comparison of different variants of the Hanson-Wright inequality. In all variants, we assume that  $(\pmb{x},\pmb{y}) = (X_i,Y_i)^n$  are subgaussian, centered, and the pairs  $(X_{i},Y_{i})$  are independent across  $i$ . We use  $\circ$  to denote elementwise multiplication, which allows us to express hard sparsity with the sparsity mask  $\gamma \in \{0,1\}^n$ . The concentration radius corresponds to the size of typical fluctuations guaranteed by the concentration inequality, i.e. the  $\epsilon$  needed for high probability guarantees.
$^{a}$  (Rudelson and Vershynin, 2013, Theorem 1.1);  $^{b}$  (Park et al., 2022, Theorem 1);  $^{c}$  (Wu and Sahai, 2023, Theorem 4)
proof of Theorem 4.1 is actually simpler than in Park et al. (2022); Wu and Sahai (2023), as bounded with soft sparsity turns out to be easier to work with than subgaussian with hard sparsity. We refer readers to Wu and Sahai (2023) for a more in-depth discussion of how these new "sparse" variants overcome the limitations of previous proof techniques used to study classification problems.
We briefly illustrate how Theorem 4.1 can be used to get tighter results throughout our analysis. A quick calculation reveals that the label vectors  $\Delta y$  and  $y_{\alpha}$  both have soft sparsity at level  $\pi = O(1 / k)$ . However,  $y_{\alpha}$  does not have hard sparsity as required by the variants in Park et al. (2022); Wu and Sahai (2023). Since  $\| M\| _F^2\leq n\| M\| _2^2$ , we obtain a concentration radius  $\epsilon$  which scales like  $\sqrt{n / k}$  rather than  $\sqrt{n}$  (obtained via vanilla Hanson-Wright) or  $n / \sqrt{k}$  (obtained via Cauchy-Schwarz). This gain is crucial to tightly analyzing the survival, contamination, and correlation structure.
4.3 Completing the proof sketch
Theorem 4.1 and the above insights about sparsity and independence allow us to prove the following bounds on the relative survival and contamination terms which are tight up to log factors; see the Appendix for more details. For brevity's sake, we introduce the notation  $\mu \triangleq n^{q + r - 1}$ .
Proposition 4.2 (Bounds on relative survival). Suppose we are in the bi-level model. With probability at least  $1 - O(1/n)$ ,
$$
\lambda_ {F} \widehat {\boldsymbol {h}} _ {\alpha , \beta} [ \alpha ] = \min  \left\{\mu^ {- 1}, 1 \right\} \Theta \left(n ^ {- \min  \left\{t, \frac {1}{2} \right\}}\right) \sqrt {\log k}.
$$
Next, we state our bounds on contamination.
Proposition 4.3 (Bounds on contamination). Suppose we are in the bi-level model. Then with probability at least  $1 - O(1/n)$ , the contamination satisfies
$$
\mathsf {C N} _ {\alpha , \beta} = \underbrace {\min  \left\{\mu^ {- 1} , 1 \right\} \Theta \left(n ^ {\frac {r - t - 1}{2}}\right)} _ {\text {f a v o r e d f e a t u r e s}} + \underbrace {\Theta \left(n ^ {\frac {1 - t - p}{2}}\right)} _ {\text {u n f a v o r e d f e a t u r e s}}. \tag {26}
$$
Translating the parameters in Propositions 4.2 and 4.3 we see that (i) the relative survival is diminished by a factor  $1 / k$  as long as  $k = o(\sqrt{n})$ , and a factor  $1 / \sqrt{n}$  for  $k = \Omega (\sqrt{n})$  (this looseness ends up being negligible for the final result) and (ii) the contamination is diminished by a factor of  $1 / \sqrt{k}$ . This essentially matches the expected behavior from the heuristic calculation in Subramanian et al. (2022). Together with some straightforward algebra, Propositions 4.2 and 4.3 allow us to compute the regimes where the survival-to-contamination ratio SU/CN grows or decays polynomially. This yields the stated regimes in Conjecture 3.1; see Appendix A.1 for more details.
For technical reasons, the analogous bounds in Subramanian et al. (2022) are loose, giving rise to unnecessary conditions for good generalization such as  $t < p - 2$  and  $t < 2q + r - 2$ . Moreover, we are able to give both upper and lower bounds on the survival and contamination terms, whereas they only give one-sided inequalities for each quantity.
In this paper we resolve the main conjecture of Subramanian et al. (2022), identifying the exact regimes where an overparameterized linear model succeeds at multiclass classification. Our techniques also lay the foundation for investigating related generalization for other multiclass tasks and nonlinear algorithms. We hope that by bringing the rigorous proofs closer to the heuristic style of calculation, we open the path for analyzing more complicated and realistic models.
An important next step is to extend our results to more realistic spiked covariance models. For example, one typically observes power-law decay for the extreme eigenvalues in applications. We expect that the bi-level model can be relaxed to allow for constant deviations in the weightings for the favored, non-label defining features and power law decay for the unfavored features. The former change would likely only affect constants in certain areas of the argument that do not crucially depend on the exact constants involved, whereas the latter would likely just change the effective degree of overparameterization (Bartlett et al., 2020). However, even constant fluctuations in weighting for the label-defining features can lead to significant subtleties, as these constant deviations manifest as polynomially large variations in the number of examples of each class. Such heterogeneity between label-defining directions would likely lead to significantly messier conditions for generalization.
Another future direction is to move beyond Gaussian features. It is plausible that similar results would hold for vector subgaussian features which are rotationally invariant, allowing us to rotate into the basis where the features have diagonal covariance. One place where Gaussianity is crucially used is to obtain an explicit lower bound on the margin between the features.
As an example application, we sketch out how our proof techniques imply precise conditions for a variant of the learning task called multilabel classification. In a simple model for multilabel classification, each datapoint can have several of  $k$  possible labels — corresponding to the positive valued features — but in the training set only one such correct label is provided at random for each datapoint. We deem that the model generalizes if for any queried label it successfully labels test inputs as positive or negative. We can use the MNI approach here to learn classifiers.
Some thought reveals that the main difference between multilabel classification and multiclass classification from a survival and contamination perspective is that positive features no longer need to outcompete other features. Thus, the main object of study would be the bilinear forms  $z_{j}^{\top}A^{-1}y_{\alpha}$ , which is possible thanks to Theorem 4.1. The survival and contamination terms are only affected by the expected values of these bilinear forms, but the expected values match the multiclass behavior up to log factors, which do not affect the regimes where SU/CN will grow or shrink polynomially. A similar analysis thus reveals that MNI will generalize in exactly the same regimes as in Conjecture 3.1. Here, the model generalizes in the sense that with high probability over the labels the model will correctly classify, and failure to generalize means that the model will do no better than a coin toss.
Perhaps surprisingly, resolving Conjecture 3.1 also implies that MNI is asymptotically suboptimal compared to a natural non-interpolative approach: simply make  $\widehat{\pmb{f}}_m$  equal to the average of all positive training examples of class  $m$ . A straightforward analysis, detailed in the supplementary material, reveals this scheme fails to generalize exactly when  $t < \min \left\{1 - r,p + 1 - 2(q + r)\right\}$ , even in the regime where regression succeeds ( $q + r < 1$ ). This is particularly interesting because we have shown that in the regime where regression succeeds, MNI generalizes only when  $t < \min \left\{1 - r,p - 1\right\}$ , which is a smaller region. In light of this gap, it would be interesting to identify the information-theoretic barrier for multiclass classification, especially within the broader context of statistical-computation gaps (see e.g. Wu and Xu (2021); Brennan and Bresler (2020)).
DW acknowledges support from NSF Graduate Research Fellowship DGE-2146752. We acknowledge funding support from NSF AST-2132700 for this work. DW appreciates helpful discussions with Sidhanth Mohanty and Prasad Raghavendra.
<sup>6</sup>Note that Frei et al. (2023) point out that even leaky ReLU networks trained with a gradient flow can behave like averages of training examples.
David X Wu and Anant Sahai. Lower bounds for multiclass classification with overparameterized linear models. In International Symposium on Information Theory, 2023.
Yihong Wu and Jiaming Xu. Statistical problems with planted structures: Information-theoretical and computational limits. Information-Theoretic Methods in Data Science, 383:13, 2021.
Mengjia Xu, Akshay Rangamani, Qianli Liao, Tomer Galanti, and Tomaso Poggio. Dynamics in deep classifiers trained with the square loss: Normalization, low rank, neural collapse, and generalization bounds. Research, 6:0024, 2023.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. arXiv preprint arXiv:1611.03530, 2016.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3):107-115, 2021.
Shuheng Zhou. Sparse Hanson-Wright inequalities for subgaussian quadratic forms. Bernoulli, 25 (3):1603-1639, 2019.
1 Introduction 1
1.1 Brief treatment of related work 2
2 Problem setup 3
3 Main results 4
4 Technical overview 6
4.1 Bounding the survival-to-contamination ratio 7
4.2 A new variant of the Hanson-Wright inequality 8
4.3 Completing the proof sketch 9
A Preliminaries and notation 17
A.1 Proof of Theorem 3.2 18
B Main tools 22
B.1 Hanson-Wright Inequality 22
B.2 Gram matrices and the Woodbury formula 22
B.3 Concentration of spectrum 24
C Utility bounds: applying the tools 25
D Bounding the survival 28
E Bounding the contamination 29
E.1 Upper bounding the contamination from label-defining+favored features 30
E.2 Lower bounding the contamination from label-defining+favored features 31
E.3 Bounding the unfavored contamination 32
F Obtaining tight misclassification rate 34
F.1 Main results for tight misclassification rates 35
F.2 Lower bounding the denominator 36
F.3 Upper bounding the numerator: the unnormalized correlation 36
F.3.1 Bounding the favored correlation 36
F.3.2 Bounding the unfavored correlation 39
G A new variant of the Hanson-Wright inequality with soft sparsity 40
G.1 Diagonal terms 40
G.2 Offdiagonal terms 41
H Proofs of main lemmas for concentration of spectrum 42
I Miscellaneous lemmas 45
J Comparison to the straightforward non-interpolative scheme 46
K Alternative framing of bi-level ensemble 47
A Preliminaries and notation
For positive integers  $n$ , we use the shorthand  $[n] \triangleq \{1, \ldots, n\}$ . For a vector  $\boldsymbol{v} \in \mathbb{R}^n$ ,  $\| \boldsymbol{v} \|_2$  always denotes the Euclidean norm. We index entries by using square brackets, so  $\boldsymbol{v}[j]$  denotes the  $j$ th entry of  $\boldsymbol{v}$ . For any matrix  $M \in \mathbb{R}^{m \times n}$ , we denote its  $ij$ th entry by  $m_{ij}$ ,  $\| M \|_2$  denotes the spectral norm, and  $\| M \|_F = \operatorname{Tr}(M^\top M)$  denotes the Frobenius norm. We use  $\sigma_{\max}(M)$  and  $\sigma_{\min}(M)$  to denote the maximum and minimum singular values of  $M$ , respectively. If  $M \in \mathbb{R}^{n \times n}$  is symmetric, we write  $\mu_1(M) \geq \mu_2(M) \geq \ldots \geq \mu_n(M)$  to denote the ordered eigenvalues of  $M$ . Given two vectors  $\boldsymbol{v}, \boldsymbol{u} \in \mathbb{R}^n$ , we write  $\boldsymbol{v} \circ \boldsymbol{u} \in \mathbb{R}^n$  to denote the entrywise product of  $\boldsymbol{v}$  and  $\boldsymbol{u}$ .