SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} F \left(\alpha_ {2}\right) = \alpha_ {2} F ^ {\prime} (0) + \frac {\alpha_ {2} ^ {2}}{2} F ^ {\prime \prime} (\eta_ {0}) \\ \leq \frac {c _ {2} \epsilon^ {4 2}}{d} \left(- \frac {\epsilon^ {1 8 . 5}}{2} + \frac {c _ {2} \epsilon^ {4 2}}{2 d} \frac {C _ {5}}{\epsilon^ {5}}\right) \\ <   - \frac {C _ {7} \epsilon^ {6 1}}{d} \text {f o r s o m e c o n s t a n t} C _ {7} > 0 \\ \end{array}
$$
Then we consider the case when  $a' > 0$ . By Lemma 23, we have either  $F'(0) < -\frac{C_2\epsilon^{36.5}}{\log^4(1/\epsilon)}$  or  $F''(0) > \frac{C_3\epsilon^{34.5}}{\log^4(1/\epsilon)}$ . Here we consider three cases with respect to  $F'(0)$ .
Case 1:  $F'(0) < -\frac{C_2 \epsilon^{36.5}}{\ln^4(1 / \epsilon)}$ . We use  $\alpha_{2}$  in this case. By Taylor expansion, there exists  $\eta_{1}$  such that  $\alpha_{2} < \eta_{1} < 0$  and
$$
F (\alpha_ {2}) = F (0) + \alpha_ {2} F ^ {\prime} (0) + \frac {\alpha_ {2} ^ {2}}{2} F ^ {\prime \prime} (\eta_ {1})
$$
By Lemma 24, we know  $|F''(\eta_1)| \leq C_5 / \epsilon^5$ . By choosing  $\alpha_{2} = -c_{2}\epsilon^{42} / d$
$$
\begin{array}{l} F (\alpha_ {1}) = \alpha_ {2} F ^ {\prime} (0) + \frac {\alpha_ {2} ^ {2}}{2} F ^ {\prime \prime} (\eta_ {1}) \\ \geq \frac {c _ {2} \epsilon^ {4 2}}{d} \left(\frac {C _ {2} \epsilon^ {3 6 . 5}}{\ln^ {4} (1 / \epsilon)} - \frac {c _ {2} \epsilon^ {4 2}}{2 d} \frac {C _ {5}}{\epsilon^ {5}}\right) \\ > \frac {C _ {8} \epsilon^ {7 9}}{d} \text {f o r s o m e c o n s t a n t} C _ {8} > 0 \\ \end{array}
$$
Case 2:  $-\frac{C_2\epsilon^{36.5}}{\ln^4(1 / \epsilon)} \leq F'(0) \leq \frac{c_s\epsilon^{77}}{d}$  with some constant  $c_{s} > 0$ . We use  $\alpha_{2}$  in this case. By Lemma 23, we know  $F''(0) > C_3\epsilon^{34.5} / \ln^4 (1 / \epsilon)$ . Then there exists  $\eta_{2}$  satisfying  $\alpha_{2} < \eta_{2} < 0$  and
$$
F (\alpha_ {2}) = F (0) + \alpha_ {2} F ^ {\prime} (0) + \frac {\alpha^ {2}}{2} F ^ {\prime \prime} (0) + \frac {\alpha_ {2} ^ {3}}{6} F ^ {\prime \prime \prime} (\eta_ {2}).
$$
By Lemma 24, we know  $|F'''(\eta_2)| \leq C_6 / \epsilon^7$ . Thus by choosing  $\alpha_{2} = -c_{2}\epsilon^{42} / d$
$$
\begin{array}{l} F \left(\alpha_ {2}\right) = \alpha_ {2} F ^ {\prime} (0) + \frac {\alpha_ {2} ^ {2}}{2} F ^ {\prime \prime} (0) + \frac {\alpha_ {2} ^ {3}}{6} F ^ {\prime \prime \prime} (\eta_ {2}) \\ > \frac {c _ {2} \epsilon^ {4 2}}{d} \left(- \frac {c _ {s} \epsilon^ {7 7}}{d} + \frac {c _ {2} \epsilon^ {4 2}}{2 d} \frac {C _ {3} \epsilon^ {3 4 . 5}}{\ln^ {4} (1 / \epsilon)} - \frac {c _ {2} ^ {2} \epsilon^ {8 4}}{6 d ^ {2}} \frac {C _ {6}}{\epsilon^ {7}}\right) \\ \geq \frac {C _ {9} \epsilon^ {1 1 9}}{d ^ {2}} \text {f o r s o m e c o n s t a n t} C _ {9} > 0 \\ \end{array}
$$
Case 3:  $F'(0) > \frac{c_s \epsilon^{77}}{d}$ . We use  $\alpha_1$  in this case. Then there exists  $\eta_3$  satisfying  $\alpha_1 < \eta_3 < 0$  and
$$
F (\alpha_ {1}) = F (0) + \alpha_ {1} F ^ {\prime} (0) + \frac {\alpha_ {1} ^ {2}}{2} F ^ {\prime \prime} (\eta_ {3}).
$$
By Lemma 24, we know  $|F''(\eta_3)| \leq C_5 / \epsilon^5$ . For  $\alpha_{1} = -c_{1}\epsilon^{82} / d$ , we have
$$
\begin{array}{l} F (\alpha_ {1}) = \alpha_ {1} F ^ {\prime} (0) + \frac {\alpha_ {1} ^ {2}}{2} F ^ {\prime \prime} (\eta_ {3}) \\ \leq \frac {c _ {1} \epsilon^ {8 2}}{d} \left(- \frac {c _ {s} \epsilon^ {7 7}}{d} + \frac {c _ {1} \epsilon^ {8 2}}{2 d} \frac {C _ {5}}{\epsilon^ {5}}\right) \\ \leq - \frac {C _ {1 0} \epsilon^ {1 5 9}}{d ^ {2}} \text {f o r s o m e c o n s t a n t} C _ {1 0} > 0 \\ \end{array}
$$
Then we know for all cases, there exists a constant  $C' = \min(C_7, C_8, C_9, C_{10})$  such that
$$
\max  \left(\left| F \left(\alpha_ {1}\right) \right|, \left| F \left(\alpha_ {2}\right) \right|\right) \geq \frac {C ^ {\prime} \epsilon^ {1 5 9}}{d ^ {2}}
$$
Finally we will lower bound  $\left(\underset {x\sim q}{\mathbb{E}}e^{\alpha x^2}\right)^{d - 1}$  as follows.
$$
\left(\underset {x \sim q} {\mathbb {E}} e ^ {\alpha x ^ {2}}\right) ^ {d - 1} \geq \left(\underset {x \sim q} {\mathbb {E}} \left(1 + \alpha x ^ {2}\right)\right) ^ {d - 1} = (1 + \alpha) ^ {d - 1} \geq \left(1 - \frac {1}{d}\right) ^ {d - 1} \geq 1 / e
$$
Let  $C = C' / e$ , and we will get
$$
\max (| u ^ {\top} \mu_ {\alpha_ {1}} |, | u ^ {\top} \mu_ {\alpha_ {2}} |) > \frac {C \epsilon^ {1 5 9}}{d ^ {2}}.
$$
C.4 Quantitative Bounds for Contrastive Covariance: Symmetric Case
Before addressing Lemma 5 which is applicable in the scenario where  $|a + b| < \epsilon^5$ , we first demonstrate that contrastive covariance works for the case where the removed band  $[a, b]$  is symmetric around the origin. That is,  $a = -b$ . In such cases, we aim to establish that there's a noticeable difference between the top two eigenvalues ( $\lambda_1$  and  $\lambda_2$ ) of the contrastive covariance matrix  $\tilde{\Sigma}$ , stated in Lemma 25. We will then extend the lemma to the near-symmetric scenario in Section C.5.
Lemma 25 (Quantitative Spectral Gap of Contrastive Covariance - Symmetric Case). Suppose  $a + b = 0$ . Choose  $\alpha_{3} = -c_{3}\epsilon^{2}$  for some constant  $c_{3} > 0$ . Then, for an absolute constant  $C$ , the top two eigenvalues  $\lambda_{1} \geq \lambda_{2}$  of the corresponding re-weighted covariance of  $P$  satisfy
We recall the definition of moment ratio as in Definition 5.
$$
\operatorname {m r} _ {q} (t) = \frac {\operatorname {v a r} _ {q _ {t}} \left(X ^ {2}\right)}{\left(\mathbb {E} _ {q _ {a}} X ^ {2}\right) ^ {2}} = \frac {M _ {0} (t) M _ {4} (t)}{M _ {2} ^ {2} (t)} - 1, \text {w h e r e} M _ {k} (t) = \int_ {t} ^ {\infty} x ^ {k} q (x) d x.
$$
For simplicity, in the remaining section, we'll use  $\mathrm{mr}(t)$  as a shorthand notation for this moment ratio. Just as in the proof of the qualitative bound in Section C.2, the difference between the first and second eigenvalues (the spectral gap) of the re-weighted covariance matrix, denoted as  $\lambda_1 - \lambda_2$ , is proportional to the difference in the moment ratio of the distribution  $q$  at 0 and  $b$ , denoted as  $\mathrm{mr}(0) - \mathrm{mr}(b)$ . To prove the quantitative result Lemma 25, our proof strategy involves several steps.
- Proving monotonicity of Moment Ratio  $\mathrm{mr}(t)$ . This property is stated in Lemma 3 and its proof is presented in Section C.2. The proof involves reducing the case of general logconcave distributions to that of exponential distributions.
- Establishing a positive spectral gap for small  $b$ . With Lemma 26, we focus on illustrating that for values of  $b$  which are relatively small (less than a certain constant), there is a guaranteed positive gap  $\mathrm{mr}(0) - \mathrm{mr}(b)$ .
- Generalizing to any  $b$  satisfying  $\int_{-b}^{b} q(x) \, dx \geq \epsilon$  in Lemma 29. The lemma will directly imply the quantitative result lemma for the symmetric case (Lemma 25).
Having demonstrated the monotonicity of the moment ratio in Section C.2, we now begin by illustrating the positive gap, denoted as  $\mathrm{mr}(0) - \mathrm{mr}(b)$ , for small  $b$  in Lemma 26. The proof relies on the properties of moments derived by the unimodality of the distribution (Lemma 27, Lemma 28).
Lemma 26 (Gap for small  $t$ ). For  $t > 0$  such that  $\int_0^t q(x)dx\leq 0.4$ , we have the following gap
$$
m r (0) - m r (t) \geq \frac {\int_ {0} ^ {t} q (x) d x}{1 0 0}.
$$
Proof. Denote  $\int_0^t q(x)dx = \nu ,\int_0^t x^2 q(x)dx = \omega$  . Then we have
$$
\begin{array}{l} \operatorname {m r} (0) - \operatorname {m r} (t) = \frac {M _ {0} (0) M _ {4} (0)}{M _ {2} ^ {2} (0)} - \frac {M _ {0} (t) M _ {4} (t)}{M _ {2} ^ {2} (t)} \\ = 2 \int_ {0} ^ {t} x ^ {4} q (x) d x + \frac {M _ {4} (t)}{2 (\omega + M _ {2} (t)) ^ {2}} - \frac {\left(\frac {1}{2} - \nu\right) M _ {4} (t)}{M _ {2} ^ {2} (t)} \\ = 2 \int_ {0} ^ {t} x ^ {4} q (x) d x + \frac {M _ {4} (t)}{2 M _ {2} ^ {2} (t)} \left(\frac {1}{\left(1 + \frac {\omega}{M _ {2} (t)}\right) ^ {2}} - 1 + 2 \nu\right) \\ \end{array}
$$
Fix  $q(t)$  and  $\nu$ , we apply Lemma 27 and have
$$
M _ {2} (t) \geq \left(\frac {1}{2} - \nu\right) t ^ {2} \left(1 + \frac {\frac {1}{2} - \nu}{q (t) t}\right)
$$
Since  $q(x)$  is monotonically decreasing, for any  $x \leq t$ ,  $q(x) \geq q(t)$ . So we have the constraint that
$$
\nu = \int_ {0} ^ {t} q (x) d x \geq t q (t)
$$
Plug into the previous inequality and we get
$$
M _ {2} (t) \geq \left(\frac {1}{2} - \nu\right) t ^ {2} \left(1 + \frac {\frac {1}{2} - \nu}{q (t) t}\right) \geq \left(\frac {1}{2} - \nu\right) t ^ {2} \left(1 + \frac {\frac {1}{2} - \nu}{\nu}\right) = \left(\frac {1}{2} - \nu\right) \frac {t ^ {2}}{2 \nu}
$$
In addition, by fixing  $v$  and  $t$ , we apply Lemma 28 and get
$$
\omega \leq \frac {\nu t ^ {2}}{3}
$$
So we know
$$
\frac {\omega}{M _ {2} (t)} \leq \frac {\frac {\nu t ^ {2}}{3}}{\left(\frac {1}{2} - \nu\right) \frac {t ^ {2}}{2 \nu}} = \frac {2 \nu^ {2}}{3 \left(\frac {1}{2} - \nu\right)}
$$
By calculation, we will get
$$
\begin{array}{l} \frac {1}{\left(1 + \frac {\omega}{M _ {2} (t)}\right) ^ {2}} - 1 + 2 \nu \geq \frac {9 \left(\frac {1}{2} - \nu\right) ^ {2} - 2 \left(\frac {1}{2} - \nu\right) \left(\frac {3}{2} - 3 \nu + 2 \nu^ {2}\right) ^ {2}}{\left(\frac {3}{2} - 3 \nu + 2 \nu^ {2}\right) ^ {2}} \\ \geq \frac {4}{9} \left(\frac {1}{2} - \nu\right) \left(9 \left(\frac {1}{2} - \nu\right) - 2 \left(\frac {3}{2} - 3 \nu + 2 \nu^ {2}\right) ^ {2}\right) \\ = \frac {4}{9} \nu \left(\frac {1}{2} - \nu\right) \left(- 8 \nu^ {3} + 2 4 \nu^ {2} - 3 0 \nu + 9\right) \\ \end{array}
$$
Let  $T(\nu) = -8\nu^3 + 24\nu^2 - 30\nu + 9$ , then we know its derivative is
$$
T ^ {\prime} (\nu) = - 2 4 \nu^ {2} + 4 8 \nu - 3 0 = - 2 4 (\nu - 1) ^ {2} - 6 <   0
$$
So  $T(\nu)$  is monotonically decreasing. Since  $\nu < 0.4$ , we know
$$
T (\nu) \geq T (0. 4) > 0. 3, \frac {1}{2} - \nu > 0. 1
$$
Plugging in and we will get
$$
\frac {1}{\left(1 + \frac {\omega}{M _ {2} (t)}\right) ^ {2}} - 1 + 2 \nu \geq \frac {4}{9} \nu \cdot 0. 1 \cdot 0. 3 > 0. 0 1 \nu
$$
Finally by Cauchy-Schwarz Inequality, we have  $M_0(t)M_4(t)\geq M_2^2 (t)$
$$
\operatorname {m r} (0) - \operatorname {m r} (t) \geq \frac {M _ {4} (t)}{2 M _ {2} ^ {2} (t)} 0. 0 1 \nu \geq \frac {\nu}{2 0 0 M _ {0} (t)} \geq \frac {\nu}{1 0 0}
$$
Lemma 27. Let  $t, r \geq 0, 0 < s \leq 1$ . Define  $\mathcal{P} = \{p(x) : [t, \infty) \to [0, 1), \text{logconcave}, p'(x) \leq 0, \int_t^\infty p(x) dx = s, p(t) = r\}$ . Then we have
$$
\min  _ {p \in \mathcal {P}} \int_ {t} ^ {\infty} x ^ {2} p (x) d x \geq s t ^ {2} \left(1 + \frac {s}{r t}\right).
$$
Proof. For any  $p \in \mathcal{P}$ , we denote  $M_2(p) = \int_t^\infty x^2 p(x) dx$ . Define
$$
u (x) = r \cdot \mathbb {1} _ {x \in [ t, t + s / r ]}
$$
Clearly  $u \in \mathcal{P}$ . We will show that  $u(x) = \operatorname*{argmin}_{p \in P} M_2(p)$ . For any  $p \in \mathcal{P}$ , we have  $\int_t^\infty u(x) \, dx = \int_t^\infty p(x) \, dx$ ,  $u(t) = p(t)$  and  $p'(x) \leq 0$ . So we know the graph of  $u$  and  $p$  intersects at points  $t$  and  $t + s/r$ , where  $u(x) \geq p(x)$  in the interval  $[t, t + s/r]$  and  $u(x) < p(x)$  outside the interval. So we know
$$
\int_ {t} ^ {t + s / r} u (x) - p (x) d x = \int_ {t + s / r} ^ {\infty} p (x) - u (x) d x
$$
Since for any  $x \in [t, t + s / r]$ , any  $y \in [t + s / r, \infty)$ , we have  $x \leq y$ . So we have
$$
M _ {2} (u) - M _ {2} (p) = \int_ {t} ^ {t + s / r} (u (x) - p (x)) x ^ {2} d x - \int_ {t + s / r} ^ {\infty} (p (x) - u (x)) x ^ {2} d x \leq 0
$$
$$
\min_{p\in \mathcal{P}}M_{2}(p) = M_{2}(u).
$$
By calculating  $M_2(u)$ , we have
$$
\min  _ {p \in \mathcal {P}} M _ {2} (p) = M _ {2} (u) = \int_ {t} ^ {t + s / r} r x ^ {2} d x = r \left(t ^ {2} \frac {s}{r} + t \frac {s ^ {2}}{r ^ {2}} + \frac {s ^ {3}}{3 r ^ {3}}\right) \geq s t ^ {2} \left(1 + \frac {s}{r t}\right)
$$
Lemma 28. Let  $t \geq 0, 0 < s \leq 1$ . Define  $\mathcal{P} = \{p(x) : [0, t] \to [0, 1), \text{logconcave}, p'(x) \leq 0, \int_0^t p(x) dx = s\}$ . Then we have
$$
\max _ {p \in \mathcal {P}} \int_ {0} ^ {t} x ^ {2} p (x) d x = \frac {s t ^ {2}}{3}.
$$
Proof. For any  $p \in \mathcal{P}$ , we denote  $M_2(p) = \int_0^t x^2 p(x) \, dx$ . Define  $u(x) = \frac{s}{t} \cdot \mathbb{1}_{x \in [0,t]}$ . Clearly,  $u \in \mathcal{P}$ . Then for any  $p \in \mathcal{P}$ , because it is monotonically decreasing and  $\int_0^t p(x) \, dx = \int_0^t u(x) \, dx$ , the graphs of  $p$  and  $u$  intersect at point  $l \in [0,t]$ . Also  $p(x) \geq u(x)$  for  $x \in [0,l]$  and  $p(x) \leq u(x)$  for  $x \in [l,t]$ . Since for any  $x \in [0,l]$  and any  $y \in [l,t]$ , we have  $x \leq y$ . So we know
$$
M _ {2} (p) - M _ {2} (u) = \int_ {0} ^ {l} x ^ {2} (p (x) - u (x)) d x - \int_ {l} ^ {t} x ^ {2} (u (x) - p (x)) \leq 0
$$
$$
\max _ {p \in \mathcal {P}} M _ {2} (p) = M _ {2} (u) = \int_ {0} ^ {t} \frac {s}{t} x ^ {2} d x = \frac {s}{t} \frac {1}{3} t ^ {3} = \frac {s t ^ {2}}{3}
$$
Using Lemma 3 and Lemma 26, we will show the positive gap  $\mathrm{mr}(0) - \mathrm{mr}(b)$  for any  $b$  satisfying  $\int_{-b}^{b} q(x) \, dx \geq \epsilon$  as in Lemma 29.
Lemma 29 (Gap for Log-concave Distribution). Let  $0 < \epsilon < 0.1$ , let  $b > 0$  satisfying  $\int_0^b q(x)dx\geq \epsilon /2$ . Then we have for  $mr(0) - mr(b)\geq \epsilon /200$ .
Proof. Let  $b_{0}$  such that  $\int_0^{b_0}q(x)dx = \epsilon /2$ . By Lemma 26, we know
$$
\operatorname {m r} (0) - \operatorname {m r} (b _ {0}) \geq \frac {\int_ {0} ^ {b _ {0}} q (x) d x}{1 0 0} \geq \frac {\epsilon}{2 0 0}
$$
By Lemma 3, we know  $\mathbf{mr}'(t) \leq 0, \forall t \geq 0$ . So for any  $b > 0$  such that  $\int_0^b q(x) dx \geq \epsilon / 2$ ,  $\mathbf{mr}(b) \leq \mathbf{mr}(a_0)$ . Therefore,
$$
\operatorname {m r} (0) - \operatorname {m r} (b) \geq \operatorname {m r} (0) - \operatorname {m r} (b _ {0}) \geq \frac {\epsilon}{2 0 0}
$$
Before moving on to the proof of the quantitative lemma, we will first present a helper lemma that can be directly applied.
Lemma 30. Define
$$
S (\alpha) := \underset {x \sim \hat {q}} {\mathbb {E}} e ^ {\alpha x ^ {2}} x ^ {2} \underset {x \sim q} {\mathbb {E}} e ^ {\alpha x ^ {2}} - \underset {x \sim q} {\mathbb {E}} e ^ {\alpha x ^ {2}} x ^ {2} \underset {x \sim \hat {q}} {\mathbb {E}} e ^ {\alpha x ^ {2}} \tag {C.3}
$$
For a given  $\alpha = -c\epsilon^s$  with  $s \geq 2$  and a certain positive constant  $c$ , it can be established that
$S(\alpha) > C\epsilon^{s + 1}$ , where  $C$  is a positive constant.
Proof. We show the lower bound of  $S(\alpha)$  using Taylor expansion.
Firstly, since  $q$  and  $\hat{q}$  are both isotropic,
$$
S (0) = \underset {x \sim \hat {q}} {\mathbb {E}} x ^ {2} - \underset {x \sim q} {\mathbb {E}} x ^ {2} = 0
$$
Secondly, we will lower bound  $|S'(0)|$  using the monotonicity of moment ratio. The variance  $\sigma_1^2$  of  $q$  restricted to  $\mathbb{R} \backslash [-b, b]$  is
$$
\begin{array}{l} S (\alpha) = \frac {2 \int_ {b / \sigma_ {1}} ^ {\infty} e ^ {\alpha x ^ {2}} x ^ {2} \sigma_ {1} p (x \sigma_ {1}) d x}{2 \int_ {b} ^ {\infty} q (x) d x} \cdot 2 \int_ {0} ^ {\infty} e ^ {\alpha x ^ {2}} q (x) d x - \frac {2 \int_ {b / \sigma_ {1}} ^ {\infty} e ^ {\alpha x ^ {2}} \sigma_ {1} p (x \sigma_ {1}) d x}{2 \int_ {b} ^ {\infty} q (x) d x} \cdot 2 \int_ {0} ^ {\infty} e ^ {\alpha x ^ {2}} x ^ {2} q (x) d x \\ = \frac {2 \int_ {b} ^ {\infty} e ^ {\alpha y ^ {2} / \sigma_ {1} ^ {2}} y ^ {2} / \sigma_ {1} ^ {2} q (y) d y}{\int_ {b} ^ {\infty} q (x) d x} \cdot \int_ {0} ^ {\infty} e ^ {\alpha x ^ {2}} q (x) d x - \frac {2 \int_ {b} ^ {\infty} e ^ {\alpha y ^ {2} / \sigma_ {1} ^ {2}} q (y) d y}{\int_ {b} ^ {\infty} q (x) d x} \cdot \int_ {0} ^ {\infty} e ^ {\alpha x ^ {2}} x ^ {2} q (x) d x \\ \end{array}
$$
$$
= \frac {\frac {2}{\sigma_ {1} ^ {2}} \int_ {b} ^ {\infty} e ^ {\alpha y ^ {2} / \sigma_ {1} ^ {2}} y ^ {2} q (y) d y \cdot \int_ {0} ^ {\infty} e ^ {\alpha x ^ {2}} q (x) d x - 2 \int_ {b} ^ {\infty} e ^ {\alpha y ^ {2} / \sigma_ {1} ^ {2}} q (y) d y \cdot \int_ {0} ^ {\infty} e ^ {\alpha x ^ {2}} x ^ {2} q (x) d x}{\int_ {b} ^ {\infty} q (x) d x}
$$
Then we can compute  $S'(0)$  as
$$
\begin{array}{l} S ^ {\prime} (0) = \frac {2}{\sigma_ {1} ^ {2} \int_ {b} ^ {\infty} q (x) d x} \left(\frac {1}{\sigma_ {1} ^ {2}} \int_ {b} ^ {\infty} y ^ {4} q (y) d y \cdot \int_ {0} ^ {\infty} q (x) d x + \int_ {b} ^ {\infty} y ^ {2} q (y) d y \cdot \int_ {0} ^ {\infty} x ^ {2} q (x) d x\right) \\ - \frac {2}{\int_ {b} ^ {\infty} q (x) d x} \left(\frac {1}{\sigma_ {1} ^ {2}} \int_ {b} ^ {\infty} y ^ {2} q (y) d y \cdot \int_ {0} ^ {\infty} x ^ {2} q (x) d x + \int_ {b} ^ {\infty} q (y) d y \cdot \int_ {0} ^ {\infty} x ^ {4} q (x) d x\right) \\ = \frac {2 M _ {0} (b) M _ {4} (b) M _ {0} (0)}{M _ {2} ^ {2} (b)} - 2 M _ {4} (0) \\ = \frac {M _ {4} (b) M _ {0} (b)}{M _ {2} ^ {2} (b)} - \frac {M _ {4} (0) M _ {0} (0)}{M _ {2} ^ {2} (0)} \\ = \operatorname {m r} (b) - \operatorname {m r} (0) \\ \end{array}
$$
The last step is because the  $q$  is isotropic. By Lemma 29,  $\mathrm{mr}(0) - \mathrm{mr}(b) \geq \epsilon / 200$ . This indicates that
$$
S ^ {\prime} (0) \leq - \epsilon / 2 0 0
$$
Next, we can upper bound  $S''(\alpha)$  for any  $\alpha \leq 0$  as
$$
\begin{array}{l} S ^ {\prime \prime} (\alpha) = \frac {2}{\int_ {b} ^ {\infty} p (x) d x} \left(\frac {1}{\sigma_ {1} ^ {6}} \int_ {b} ^ {\infty} e ^ {\alpha y ^ {2} / \sigma_ {1} ^ {2}} y ^ {6} q (y) d y \cdot \int_ {0} ^ {\infty} e ^ {\alpha x ^ {2}} q (x) d x \right. \\ + \frac {1}{\sigma_ {1} ^ {4}} \int_ {b} ^ {\infty} e ^ {\alpha y ^ {2} / \sigma_ {1} ^ {2}} y ^ {4} q (y) d y \cdot \int_ {0} ^ {\infty} e ^ {\alpha x ^ {2}} x ^ {2} q (x) d x \\ - \frac {1}{\sigma_ {1} ^ {2}} \int_ {b} ^ {\infty} e ^ {\alpha y ^ {2} / \sigma_ {1} ^ {2}} y ^ {2} q (y) d y \cdot \int_ {0} ^ {\infty} e ^ {\alpha x ^ {2}} x ^ {4} q (x) d x \\ \left. - \int_ {b} ^ {\infty} e ^ {\alpha y ^ {2} / \sigma_ {1} ^ {2}} q (y) d y \cdot \int_ {0} ^ {\infty} e ^ {\alpha x ^ {2}} x ^ {6} q (x) d x\right) \\ \geq - \frac {2}{M _ {0} (b)} \left(\frac {M _ {2} (b) M _ {4} (0)}{M _ {2} (0)} + M _ {0} (b) M _ {6} (0)\right) \\ \geq - \frac {2}{M _ {0} (b)} \left(M _ {4} (0) + M _ {0} (0) M _ {6} (0)\right) \\ \end{array}
$$
By Cauchy-Schwarz Inequality,  $M_1(0) \leq \sqrt{M_2(0)M_0(0)} = 1/2$ . By Lemma 12,
$$
M _ {k} (0) \leq (2 k) ^ {k} (2 M _ {1} (0)) ^ {k} / 2 \leq (2 k) ^ {k} / 2
$$
Since  $M_0(b) \geq \epsilon$ , for some positive constant  $c_{sec}$ ,
$$
S ^ {\prime \prime} (\alpha) \geq - \frac {c _ {s e c}}{\epsilon}
$$
By Taylor expansion, we know for  $\alpha_{3} = -c\epsilon^{s}$ ,  $s \geq 2$ ,  $c = 1 / (101c_{sec})$ , there exists  $\alpha' \in [\alpha, 0]$  such that for some constant  $C > 0$ ,
$$
S (\alpha) = S (0) + \alpha S ^ {\prime} (0) + \frac {\alpha^ {2}}{2} S ^ {\prime \prime} (\alpha^ {\prime}) \geq 0 + c \epsilon^ {s} \frac {\epsilon}{2 0 0} - \frac {c ^ {2} \epsilon^ {2 s}}{2} \frac {c _ {s e c}}{\epsilon} > C \epsilon^ {s + 1}
$$
Now we are ready to prove the contrastive covariance lemma (Lemma 25).
Proof of Lemma 25. Define
Then for  $2 \leq j \leq d$ ,
$$
\begin{array}{l} \underset {x \sim P} {\mathbb {E}} e ^ {\alpha \| x \| ^ {2}} x _ {1} ^ {2} - \underset {x \sim P} {\mathbb {E}} e ^ {\alpha \| x \| ^ {2}} x _ {j} ^ {2} \\ = \left(\underset {x _ {1} \sim \hat {q}} {\mathbb {E}} e ^ {\alpha x _ {1} ^ {2}} x _ {1} ^ {2} \underset {x _ {2} \sim q} {\mathbb {E}} e ^ {\alpha x _ {2} ^ {2}} - \underset {x _ {2} \sim \hat {q}} {\mathbb {E}} e ^ {\alpha x _ {2} ^ {2}} x _ {1} ^ {2} \underset {x _ {1} \sim q} {\mathbb {E}} e ^ {\alpha x _ {1} ^ {2}}\right) \left(\underset {x \sim q} {\mathbb {E}} e ^ {\alpha x ^ {2}}\right) ^ {d - 2} \\ = S(\alpha)\frac{\underset{x\sim P}{\mathbb{E}}e^{\alpha\|x\|^{2}}x_{1}^{2}}{\underset{x_{1}\sim\hat{q}}{\mathbb{E}}e^{\alpha x_{1}^{2}}x_{1}^{2}\underset{x_{2}\sim q}{\mathbb{E}}e^{\alpha x_{2}^{2}}} \\ \end{array}
$$
Since  $\alpha_{3} < 0$ , we have
$$
\underset {x _ {1} \sim \hat {q}} {\mathbb {E}} e ^ {\alpha_ {3} x _ {1} ^ {2}} x _ {1} ^ {2} \leq \underset {x _ {1} \sim \hat {q}} {\mathbb {E}} x _ {1} ^ {2} = 1, \quad \underset {x _ {2} \sim q} {\mathbb {E}} e ^ {\alpha_ {3} x _ {2} ^ {2}} \leq 1
$$
By Lemma 30,  $\alpha_{3} = c_{3}\epsilon^{2}$  implies that  $S(\alpha_3) > C\epsilon^3$  . So we have
$$
\underset {x \sim P} {\mathbb {E}} e ^ {\alpha \| x \| ^ {2}} x _ {1} ^ {2} - \underset {x \sim P} {\mathbb {E}} e ^ {\alpha \| x \| ^ {2}} x _ {j} ^ {2} \geq C \epsilon^ {3} \underset {x \sim P} {\mathbb {E}} e ^ {\alpha \| x \| ^ {2}} x _ {1} ^ {2}
$$
Finally we will show that the first eigenvector corresponds to  $e_1$ . For any  $v \in \mathbb{R}$ , define  $\phi(v)$  as
$$
\phi (v) := \underset {x \sim P} {\mathbb {E}} \frac {e ^ {\alpha_ {3} \| x \| ^ {2}} v ^ {\top} x x ^ {\top} v}{v ^ {\top} v}
$$
Then we know for  $2 \leq j \leq d$ ,
$$
\phi \left(e _ {1}\right) - \phi \left(e _ {j}\right) > C \epsilon^ {3} \phi \left(e _ {1}\right)
$$
$$
\begin{array}{l} \phi (v) = \frac {1}{\sum_ {i = 1} ^ {d} \gamma_ {i} ^ {2}} \mathbb {E} e ^ {\alpha_ {3} \| x \| ^ {2}} \left(\sum_ {i = 1} ^ {d} \gamma_ {i} x _ {i}\right) ^ {2} \\ = \frac {1}{\sum_ {i = 1} ^ {d} \gamma_ {i} ^ {2}} \left(\sum_ {i = 1} ^ {d} \gamma_ {i} ^ {2} \phi \left(e _ {i}\right) + 2 \mathbb {E} e ^ {\alpha \| x \| ^ {2}} \sum_ {i \neq j} \gamma_ {i} \gamma_ {j} x _ {i} x _ {j}\right) \\ = \frac {1}{\sum_ {i = 1} ^ {d} \gamma_ {i} ^ {2}} \left(\sum_ {i = 1} ^ {d} \gamma_ {i} ^ {2} \phi \left(e _ {i}\right) + 2 \sum_ {i \neq j} \gamma_ {i} \gamma_ {j} \mathbb {E} e ^ {\alpha_ {3} \sum_ {k \neq i, j} x _ {k} ^ {2}} \mathbb {E} e ^ {\alpha_ {3} \langle x, e _ {i} \rangle^ {2}} x _ {1} \mathbb {E} e ^ {\alpha \langle x, e _ {j} \rangle^ {2}} x _ {j}\right) \\ = \frac {\sum_ {i = 1} ^ {d} \gamma_ {i} ^ {2} \phi (e _ {i})}{\sum_ {i = 1} ^ {d} \gamma_ {i} ^ {2}} \\ \leq \phi (e _ {1}) \\ \end{array}
$$
This shows that the top eigenvalue of  $\tilde{\Sigma}$  is  $\lambda_{1} = \max_{v}\phi (v) = \phi (e_{1})$ . In other words, the top eigenvector is  $e_1$ , which is essentially  $u$ . Similarly the second eigenvalue of  $\tilde{\Sigma}$  is  $\lambda_{2} = \max_{v:v\perp e_{1}}\phi (v) = \phi (e_{j}), 2\leq j\leq d$ . So we get
$$
\lambda_ {1} - \lambda_ {2} > C \epsilon^ {3} \lambda_ {1}
$$
C.5 Quantitative Bounds for Contrastive Covariance: Near-Symmetric Case
We have shown the result for symmetric case  $a + b = 0$  in Section C.4. Here we will show that we can extend the contrastive covariance lemma (Lemma 25) to the near-symmetric case, where  $|a + b| < \epsilon^5$ . In this section, we will present the proof of Lemma 5, which addresses the nearly symmetric case quantitatively. The proof idea is to approximate the re-weighted covariance of the distribution with margin  $[a,b]$ , by comparing it to the same distribution truncated with the symmetric interval  $[-b,b]$ . This enables us to generalize the result from the symmetric scenario to the near-symmetric scenario.
Recall that  $\tilde{q}$  is the distribution obtained by restricting  $q$  to the set  $\mathbb{R} \setminus [a, b]$ . We denote  $\tilde{r}$  as the distribution that is obtained by restricting  $q$  to the set  $\mathbb{R} \setminus [-b, b]$ , and  $\hat{r}$  as the isotropized distribution of  $\tilde{r}$ . Let  $\sigma_2^2$  be the variance of  $\tilde{r}$ .
To approximate the characteristics of  $\hat{q}$  using those of  $\hat{r}$ , we undertake the subsequent steps.
- Assessing the mean. We illustrate that the mean of  $\tilde{q}$  is adequately small in Lemma 31.
- Variance approximation. We approximate the variance of  $\hat{q}$  by using the variance of  $\hat{r}$ , as elaborated in Lemma 32.
- Re-weighted second moment. We use the re-weighted second moment of  $\hat{r}$  to approximate the corresponding moment in  $\hat{q}$ . The details are provided in Lemma 34.
- Re-weighted zeroth moment. We use the re-weighted zero moment of  $\hat{r}$  to approximate the corresponding moment in  $\hat{q}$ , which is shown in Lemma 35.
Lemma 31. For an integer  $s \geq 1$ , if  $|a + b| < \epsilon^{s+1}$ , then
(1) the mean of  $\tilde{q}, |\mu_1| < \epsilon^s\ln (1 / \epsilon)$ ;
(2)  $a < 0$
Proof. We first consider the case when  $a < 0$ . By Lemma 8, we have