SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\mathrm {m r} _ {q} (t) = \frac {\operatorname {v a r} _ {q _ {t}} \left(X ^ {2}\right)}{\left(\mathbb {E} _ {q _ {t}} X ^ {2}\right) ^ {2}} = \frac {\frac {\int_ {t} ^ {\infty} x ^ {4} q (x) d x}{\int_ {t} ^ {\infty} q (x) d x}}{\left(\frac {\int_ {t} ^ {\infty} x ^ {2} q (x) d x}{\int_ {t} ^ {\infty} q (x) d x}\right) ^ {2}} - 1 = \frac {M _ {0} (t) M _ {4} (t)}{M _ {2} (t) ^ {2}} - 1
$$
Next we will show that  $\mathrm{mr}'(t) < 0, \forall t \geq 0$ . By taking the derivative,
$$
\mathrm {m r} ^ {\prime} (t) = \frac {- q (t)}{M _ {2} (t)} \left(t ^ {4} M _ {0} (t) M _ {2} (t) + M _ {4} (t) M _ {2} (t) - 2 t ^ {2} M _ {0} (t) M _ {4} (t)\right)
$$
Define  $H(t) = t^4 M_0(t)M_2(t) + M_4(t)M_2(t) - 2t^2 M_0(t)M_4(t)$ . We will show that  $H(t) > 0, \forall t \geq 0$ . Clearly  $H(0) = M_4(0)M_2(0) > 0$ , So we only consider  $t > 0$  in the following proof.
By the logconcavity of  $q$ , the graph of  $h$  intersects with the graph of  $q$  at exactly two points  $u' < v$  where  $v > 0$ . Also we have  $h(x) \leq q(x)$  at the interval  $[u', v]$  and  $h(x) > q(x)$  outside the interval.
Let  $u = \max \{0, u'\}$ . So for  $x \geq 0$ ,  $(x - u)(x - v)$  has the same sign as  $h(x) - q(x)$ . Since  $t \geq 0$ , we have
Expanding and we get
This shows that  $N_4(t) \geq M_4(t)$ . To show that  $H(t) > 0$ , we consider two cases.
Secondly if  $M_2(t) - 2t^2 < 0$ , by calculation of the exponential function's moments (Lemma 18), we have
$$
\begin{array}{l} H (t) = t ^ {4} M _ {0} (t) M _ {2} (t) + M _ {4} (t) \left(M _ {2} (t) - 2 t ^ {2}\right) \\ = t ^ {4} N _ {0} (t) N _ {2} (t) + M _ {4} (t) \left(M _ {2} (t) - 2 t ^ {2}\right) \\ \geq - N _ {4} (t) \left(N _ {2} (t) - 2 t ^ {2}\right) + M _ {4} (t) \left(M _ {2} (t) - 2 t ^ {2}\right) \\ = \left(M _ {2} (t) - 2 t ^ {2}\right) \left(M _ {4} (t) - N _ {4} (t)\right) \\ \geq 0 \\ \end{array}
$$
The equality holds if and only if  $M_4(t) = N_4(t)$ . This implies that
Combining both cases,  $\mathrm{mr}'(t) < 0, \forall t \geq 0$ , which implies that the moment ratio of  $q$  is strictly decreasing with respect to  $t$ .
Contrastive Covariance. View the spectral gap of the re-weighted covariance, denoted as  $\lambda_1(\tilde{\Sigma}) - \lambda_2(\tilde{\Sigma})$ , as a function of  $\alpha$ , which we denote as  $S(\alpha)$ . By calculation,  $S(0) = 0$  and  $S'(0)$  is proportional to  $\mathrm{mr}(b) - \mathrm{mr}(0)$ , which is negative by the monotonicity property of moment ratio. Then we can prove Lemma 2 using Taylor expansion.
Proof. Denote  $M_{k}(t) = \int_{t}^{\infty}x^{k}q(x)dx$ . The variance  $\sigma_1^2$  of  $q$  restricted to  $\mathbb{R}\backslash [-b,b]$  is
$$
\sigma_ {1} ^ {2} = \frac {\int_ {b} ^ {\infty} x ^ {2} p (x) d x}{\int_ {b} ^ {\infty} p (x) d x} = \frac {M _ {2} (b)}{M _ {0} (b)}
$$
Since  $\hat{q}$  is isotropic, the density on the support  $\mathbb{R} \backslash [-b / \sigma_1, b / \sigma_1]$  is
$$
\mathbb {P} _ {\hat {q}} (x) = \frac {\sigma_ {1} q (x \sigma_ {1})}{2 \int_ {b} ^ {\infty} q (x) d x}
$$
$$
S (\alpha) := \underset {x \sim \hat {q}} {\mathbb {E}} e ^ {\alpha x ^ {2}} x ^ {2} \underset {x \sim q} {\mathbb {E}} e ^ {\alpha x ^ {2}} - \underset {x \sim q} {\mathbb {E}} e ^ {\alpha x ^ {2}} x ^ {2} \underset {x \sim \hat {q}} {\mathbb {E}} e ^ {\alpha x ^ {2}}
$$
Since  $q$  and  $\hat{q}$  are both isotropic,  $S(0) = \underset {x\sim \hat{q}}{\mathbb{E}}x^{2} - \underset {x\sim q}{\mathbb{E}}x^{2} = 0$ . Then,
$$
S ^ {\prime} (0) = \frac {M _ {4} (b) M _ {0} (b)}{M _ {2} ^ {2} (b)} - \frac {M _ {4} (0) M _ {0} (0)}{M _ {2} ^ {2} (0)}
$$
The last step is because the distribution  $q$  is isotropic. By Lemma 3, we know  $S'(0) = \mathrm{mr}'(0) < 0$ . On the other hand,  $\forall \alpha \leq 0$ ,  $S''(\alpha)$  can be bounded.
$$
S ^ {\prime \prime} (\alpha) \leq \frac {M _ {6} (b)}{M _ {0} (b) M _ {2} ^ {3} (b)} + \frac {M _ {4} (b)}{M _ {0} (b) M _ {2} ^ {2} (b)} <   \operatorname {p o l y} (1 / \epsilon)
$$
By Taylor expansion, we know there exists  $\alpha < 0$  such that
$$
S (\alpha) = S (0) + \alpha S ^ {\prime} (0) + \frac {\alpha^ {2}}{2} S ^ {\prime \prime} \left(\alpha^ {\prime}\right) > 0, \text {w h e r e} \alpha^ {\prime} \in [ \alpha , 0 ]
$$
Then we have for  $2 \leq j \leq d$ ,
$$
\underset {x \sim P} {\mathbb {E}} e ^ {\alpha \| x \| ^ {2}} x _ {1} ^ {2} - \underset {x \sim P} {\mathbb {E}} e ^ {\alpha \| x \| ^ {2}} x _ {j} ^ {2} = S (\alpha) \left(\underset {x \sim q} {\mathbb {E}} e ^ {\alpha x ^ {2}}\right) ^ {d - 2} > 0
$$
For any  $v\in \mathbb{R}^d$  , define  $\phi (v)$  as
$$
\phi (v) := \underset {x \sim P} {\mathbb {E}} \frac {e ^ {\alpha \| x \| ^ {2}} v ^ {\top} x x ^ {\top} v}{v ^ {\top} v}
$$
Substituting  $v$  with  $e_1$  and  $e_j$ ,  $2 \leq j \leq d$ , we have
$$
\phi (e _ {1}) = \underset {x \sim P} {\mathbb {E}} e ^ {\alpha \| x \| ^ {2}} x _ {1} ^ {2}, \phi (e _ {j}) = \underset {x \sim P} {\mathbb {E}} e ^ {\alpha \| x \| ^ {2}} x _ {j} ^ {2}
$$
This implies that for  $2 \leq j \leq d$ ,
$$
\phi (e _ {1}) - \phi (e _ {j}) = \underset {x \sim P} {\mathbb {E}} e ^ {\alpha \| x \| ^ {2}} x _ {1} ^ {2} - \underset {x \sim P} {\mathbb {E}} e ^ {\alpha \| x \| ^ {2}} x _ {j} ^ {2} > 0
$$
For any vector  $v = \sum_{i=1}^{d} \gamma_i e_i$ , we have
$$
\phi (v) = \frac {1}{\sum_ {i = 1} ^ {d} \gamma_ {i} ^ {2}} \mathbb {E} e ^ {\alpha \| x \| ^ {2}} \left(\sum_ {i = 1} ^ {d} \gamma_ {i} x _ {i}\right) ^ {2} = \frac {\sum_ {i = 1} ^ {d} \gamma_ {i} ^ {2} \phi (e _ {i})}{\sum_ {i = 1} ^ {d} \gamma_ {i} ^ {2}} \leq \phi (e _ {1})
$$
This shows that the top eigenvalue of  $\tilde{\Sigma}$  is  $\lambda_1(\tilde{\Sigma}) = \max_v\phi (v) = \phi (e_1)$ . Similarly,  $\lambda_{2}(\tilde{\Sigma}) = \max_{v:v\perp e_{1}}\phi (v) = \phi (e_{j}), 2\leq j\leq d$ . Therefore,  $\lambda_1(\tilde{\Sigma}) > \lambda_2(\tilde{\Sigma})$  and the top eigenvector is  $e_1$ , which is essentially  $u$ .
C.3 Quantitative Bounds for Contrastive Mean
We will prove Lemma 4 in this section. Here we consider the case when  $|a + b| \geq \epsilon^5$ . We compute the contrastive mean of  $P$  given  $\alpha < 0$  as  $\mathbb{E}_{x \sim P} e^{\alpha x^2} x$  using two different  $\alpha$ 's.
Definition 6. We define  $F(\alpha)$  as re-weighted mean for the one-dimensional distribution  $\hat{q}$ .
$$
F (\alpha) = \underset {x \sim \hat {q}} {\mathbb {E}} e ^ {\alpha x ^ {2}} x = \int_ {\mathbb {R} \backslash \left[ a ^ {\prime}, b ^ {\prime} \right]} e ^ {\alpha x ^ {2}} x \hat {q} (x) d x \tag {C.2}
$$
Since  $P$  is isotropic,  $F(0) = \underset {x\sim \hat{q}}{\mathbb{E}}x = 0$
To prove Lemma 4, we need to show that for given  $\alpha_{1},\alpha_{2}$ , the maximum of  $|F(\alpha_1)|,|F(\alpha_2)|$  exceeds a certain positive threshold. We follow the same idea of bounding the number of roots of  $F(\alpha)$  as in the qualitative lemma (Lemma 1). By taking the derivative of  $F(\alpha)$ , we can show that either  $F^{\prime}(0)\neq 0$  or  $F''(0)\neq 0$ . Then by Taylor expansion, we can choose two distinct  $\alpha$ 's (near zero) so that one of the corresponding contrastive means is bounded away from zero.
In the process of proving the quantitative bounds, similar to our approach with qualitative bounds, we must consider two distinct scenarios based on the sign of  $a'$ , as illustrated in Figure 4.1.
- In the case where  $a'$  is negative, Lemma 20 asserts that the first derivative of  $F$  at zero,  $F'(0)$ , is always positive.
- Conversely, when  $a'$  is nonnegative, Lemma 23 reveals an essential characteristic of the function  $F(\alpha)$ : it's not possible for both  $F'(0)$  and  $F''(0)$  to be zero at the same time.
- Lemma 24 provides upper bounds for the derivatives of  $F(\alpha)$ . These upper bounds are crucial as they help in managing the extra terms that emerge during the Taylor expansion of  $F(\alpha)$ .
- The section concludes with the proof of Lemma 4, which is the quantitative lemma for the contrastive mean.
We start with Lemma 19 showing that  $|\mu_1|$  is away from zero provided that  $|a + b|$  is also different from zero.
Lemma 19 (Lower Bound of  $|\mu_1|)$ . If  $|a + b| \geq \epsilon^s$  for  $s \geq 2$ , then  $|\mu_1| \geq \epsilon^s / 2e$ .
Proof. Firstly, let's consider the case when  $a \leq 0$ . By Lemma 9, we know  $q(x)$  is upper bounded by 1. Since  $|a + b| \geq \epsilon^s$  and  $q$  is logconcave, by Lemma 8, the mean of the density restricted  $q$  in  $[-a,b]$  satisfies  $\mu_{[-a,b]} \geq 1 / e$ . Then
$$
| \mu_ {1} | = \frac {\mu_ {[ - a , b ]} \int_ {- a} ^ {b} q (x) d x}{1 - \int_ {a} ^ {b} q (x) d x} \geq \frac {1}{e} \frac {\int_ {- a} ^ {b} q (x) d x}{\int_ {- a} ^ {b} q (x) d x + 2 \int_ {b} ^ {\infty} q (x) d x}
$$
By Lemma 11,
$$
q (b) \geq 2 \int_ {b} ^ {\infty} q (x) d x
$$
Since  $|a| < |b|$ , we have
$$
\int_ {- a} ^ {b} q (x) d x \geq (b + a) q (b) \geq 2 (b + a) \int_ {b} ^ {\infty} q (x) d x
$$
$$
| \mu_ {1} | \geq \frac {1}{e} \frac {2 (b + a)}{2 (b + a) + 2} \geq \frac {1}{e} \frac {\epsilon^ {s}}{1 + \epsilon^ {s}} > \frac {\epsilon^ {s}}{2 e}
$$
Secondly, when  $a > 0$ , since  $b - a > \epsilon$
$$
| \mu_ {1} | = \frac {\mu_ {[ a , b ]} \int_ {a} ^ {b} q (x) d x}{1 - \int_ {a} ^ {b} q (x) d x} > (a + \frac {\epsilon}{e}) \frac {\epsilon}{1 - \epsilon} > \frac {\epsilon^ {2}}{e}
$$
Lemma 20 (Derivative of  $F(0)$  when  $a' < 0$ ). If  $|a + b| \geq \epsilon^s$  for  $s \geq 2$  and  $a' < 0$ , then  $F'(0) > \epsilon^{3s + 3.5} / 2$ .
Proof. We rearrange  $F(\alpha)$  by combining terms with same  $e^{\alpha x^2}$  as in Figure 4.1, and get
Define  $r(x) = \left\{ \begin{array}{ll} - \hat{q} (-x) & x\in [-a',b']\\ \hat{q} (x) - \hat{q} (-x) & x\in [b',\infty) \end{array} \right.$  Then we have
$$
F (\alpha) = \int_ {- a ^ {\prime}} ^ {\infty} x r (x) e ^ {\alpha x ^ {2}} d x
$$
By calculating the derivative of  $F(\alpha)$ , we have
$$
\begin{array}{l} F ^ {\prime} (\alpha) = \int_ {- a ^ {\prime}} ^ {\infty} x ^ {3} r (x) e ^ {\alpha x ^ {2}} d x \\ = \int_ {- a ^ {\prime}} ^ {\infty} x \left(x ^ {2} - b ^ {\prime 2}\right) r (x) e ^ {\alpha x ^ {2}} d x + b ^ {\prime 2} \int_ {- a ^ {\prime}} ^ {\infty} x r (x) e ^ {\alpha x ^ {2}} d x \\ \end{array}
$$
$$
= \int_ {- a ^ {\prime}} ^ {\infty} x (x ^ {2} - b ^ {\prime 2}) r (x) e ^ {\alpha x ^ {2}} d x + b ^ {\prime 2} F (\alpha)
$$
Since  $r(x)$  is nonnegative for  $x \geq b'$  and negative otherwise, then for any  $x \geq -a'$ , we have
$$
x \left(x ^ {2} - b ^ {\prime 2}\right) r (x) e ^ {\alpha x ^ {2}} \geq 0
$$
Since  $F(0) = 0$ , we have
$$
F ^ {\prime} (0) = \int_ {- a ^ {\prime}} ^ {\infty} x \left(x ^ {2} - b ^ {\prime 2}\right) r (x) d x \geq \int_ {- a ^ {\prime}} ^ {b ^ {\prime}} x \left(b ^ {\prime 2} - x ^ {2}\right) \hat {q} (- x) d x
$$
By calculation,
$$
\begin{array}{l} F ^ {\prime} (0) \geq \frac {1}{1 - \int_ {a} ^ {b} q (x) d x} \int_ {- \frac {a - \mu_ {1}}{\sigma_ {1}}} ^ {\frac {b - \mu_ {1}}{\sigma_ {1}}} x \left(\left(\frac {b - \mu_ {1}}{\sigma_ {1}}\right) ^ {2} - x ^ {2}\right) \sigma_ {1} q (- x \sigma_ {1} + \mu_ {1}) d x \\ \geq \frac {1}{\sigma_ {1} ^ {3} \left(1 - \int_ {a} ^ {b} q (x) d x\right)} \int_ {- a} ^ {b - 2 \mu_ {1}} \left(x + \mu_ {1}\right) \left(\left(b - \mu_ {1}\right) ^ {2} - \left(x + \mu_ {1}\right) ^ {2}\right) q (x) d x \\ \geq \frac {1}{\sigma_ {1} ^ {3} \left(1 - \int_ {a} ^ {b} q (x) d x\right)} \int_ {- a} ^ {b} (x + \mu_ {1}) (x + b) (b - x - 2 \mu_ {1}) q (x) d x \\ \geq \frac {1}{\sigma_ {1} ^ {3} \left(1 - \int_ {a} ^ {b} q (x) d x\right)} (b - a) 2 | \mu_ {1} | \int_ {- a} ^ {b} (x + \mu_ {1}) q (x) d x \\ \end{array}
$$
Choose  $t_0 \in [-a, b]$  such that  $\int_{-a}^{t_0} q(x) \, dx = \int_{t_0}^b q(x) \, dx$ . Since  $q(x)$  is bounded by 1 by Lemma 9, we have
$$
t _ {0} + a \geq \frac {\int_ {- a} ^ {b} q (x) d x}{2}
$$
On the other hand, similar to the proof of Lemma 19, we have
$$
\int_ {- a} ^ {b} q (x) d x \geq 2 (b + a) \int_ {b} ^ {\infty} q (x) d x \geq 2 \epsilon^ {s + 1}
$$
$$
\begin{array}{l} \int_ {- a} ^ {b} (x + \mu_ {1}) q (x) d x \geq \int_ {t _ {0}} ^ {b} (x + a) q (x) d x \geq (t _ {0} + a) \frac {1}{2} \int_ {- a} ^ {b} q (x) d x \\ \geq \frac {1}{4} \left(\int_ {- a} ^ {b} q (x) d x\right) ^ {2} \geq \epsilon^ {2 s + 2} \\ \end{array}
$$
$$
\sigma_ {1} ^ {2} \leq \frac {\int_ {\mathbb {R} \backslash [ a , b ]} x ^ {2} q (x) d x}{1 - \int_ {a} ^ {b} q (x) d x} \leq \frac {\underset {x \sim q} {\mathbb {E}} x ^ {2}}{1 - \int_ {a} ^ {b} q (x) d x} \leq \frac {1}{1 - \int_ {a} ^ {b} q (x) d x}
$$
Applying Lemma 19, we know  $|\mu_1| > \epsilon^s / 2e$ . Using these results to estimate  $F'(0)$ , we get
$$
F ^ {\prime} (0) \geq \left(1 - \int_ {a} ^ {b} q (x) d x\right) ^ {0. 5} \cdot \epsilon \cdot \frac {\epsilon^ {s}}{e} \epsilon^ {2 s + 2} > \frac {\epsilon^ {3 s + 3 . 5}}{2}
$$
Lemma 21. If  $a \geq \mu_1, \mu_1 \leq 0$ , then we have  $\int_{a - 2\mu_1}^b xq(x) dx \geq |\mu_1| \int_b^\infty q(x) dx$ .
Proof. Firstly we will show that  $\int_{2\mu_1 - a}^a (x - \mu_1)q(x)dx\geq 0$ . For  $x\in [\mu_1,a]$ , we have  $(2\mu_{1} - x)^{2} - x^{2} = 4\mu_{1}(\mu_{1} - x)\geq 0$ . Since  $q(x)$  is symmetric and uni-modal, we have  $q(2\mu_1 - x)\leq q(x),\forall x\in [\mu_1,a]$ .
$$
\int_ {2 \mu_ {1} - a} ^ {a} (x - \mu_ {1}) q (x) d x = \int_ {2 \mu_ {1} - a} ^ {\mu} (x - \mu_ {1}) q (x) d x + \int_ {\mu_ {1}} ^ {a} (x - \mu_ {1}) q (x) d x
$$
$$
\begin{array}{l} = \int_ {\mu_ {1}} ^ {a} (\mu_ {1} - x) q (2 \mu_ {1} - x) d x + \int_ {\mu_ {1}} ^ {a} (x - \mu_ {1}) q (x) d x \\ = \int_ {\mu_ {1}} ^ {a} (x - \mu_ {1}) (q (x) - q (2 \mu_ {1} - x)) d x \\ \geq 0 \\ \end{array}
$$
And then, we have
$$
\begin{array}{l} \int_ {(- \infty , 2 \mu_ {1} - b ] \cup [ b - 2 \mu_ {1}, \infty)} (x - \mu_ {1}) q (x) d x = - \mu_ {1} \int_ {(- \infty , 2 \mu_ {1} - b ] \cup [ b - 2 \mu_ {1}, \infty)} q (x) d x \\ = - 2 \mu_ {1} \int_ {b - 2 \mu_ {1}} ^ {\infty} q (x) d x \\ \end{array}
$$
Since  $\mu_{1}$  is the mean of the distribution  $\tilde{q}$ . We have
$$
\int_ {\mathbb {R} \backslash [ a, b ]} (x - \mu_ {1}) q (x) d x = 0
$$
$$
\int_ {2 \mu_ {1} - b} ^ {2 \mu_ {1} - a} (x - \mu_ {1}) q (x) d x + \int_ {b} ^ {b - 2 \mu_ {1}} (x - \mu_ {1}) q (x) d x \leq 2 \mu_ {1} \int_ {b - 2 \mu_ {1}} ^ {\infty} q (x) d x
$$
$$
\begin{array}{l} 0 \geq \int_ {2 \mu_ {1} - b} ^ {2 \mu_ {1} - a} (x - \mu_ {1}) q (x) d x + \int_ {b} ^ {b - 2 \mu_ {1}} (x - \mu_ {1}) q (x) d x - 2 \mu_ {1} \int_ {b - 2 \mu_ {1}} ^ {\infty} q (x) d x \\ = - \int_ {a - 2 \mu_ {1}} ^ {b - 2 \mu_ {1}} x q (x) d x - \mu_ {1} \int_ {a - 2 \mu_ {1}} ^ {b - 2 \mu_ {1}} q (x) d x + \int_ {b} ^ {b - 2 \mu_ {1}} x q (x) d x - \mu_ {1} \int_ {b} ^ {b - 2 \mu_ {1}} q (x) d x \\ - 2 \mu_ {1} \int_ {b - 2 \mu_ {1}} ^ {\infty} q (x) d x \\ = - \int_ {a - 2 \mu_ {1}} ^ {b - 2 \mu_ {1}} x q (x) d x - \mu_ {1} \int_ {a - 2 \mu_ {1}} ^ {\infty} q (x) d x + \int_ {b} ^ {b - 2 \mu_ {1}} x q (x) d x - \mu_ {1} \int_ {b} ^ {\infty} q (x) d x \\ \end{array}
$$
This derives that
$$
\begin{array}{l} \int_ {a - 2 \mu_ {1}} ^ {b} x q (x) d x \geq - \mu_ {1} \int_ {a - 2 \mu_ {1}} ^ {\infty} q (x) d x - \mu_ {1} \int_ {b} ^ {\infty} q (x) d x \\ \geq - \mu_ {1} \int_ {b} ^ {\infty} q (x) d x \\ \end{array}
$$
Lemma 22 (Second Derivative of  $F(0)$  when  $a' \geq 0$ ). If  $a' \geq 0$ , we define the following functions,
$$
r (x) = \left\{ \begin{array}{l l} \hat {q} (x) - \hat {q} (- x) & x \in [ 0, a ^ {\prime} ] \cup [ b ^ {\prime}, \infty) \\ - \hat {q} (- x) & x \in (a ^ {\prime}, b ^ {\prime}) \end{array} \right., H (\alpha) = \int_ {0} ^ {\infty} x (x ^ {2} - a ^ {\prime 2}) (x ^ {2} - b ^ {\prime 2}) r (x) e ^ {\alpha x ^ {2}} d x.
$$
$$
F ^ {\prime \prime} (\alpha) = H (\alpha) + \left(a ^ {\prime 2} + b ^ {\prime 2}\right) F ^ {\prime} (\alpha) + a ^ {\prime 2} b ^ {\prime 2} F (\alpha).
$$
By the definition of  $r(x)$ , we naturally have
$$
F (\alpha) = \int_ {0} ^ {\infty} x r (x) e ^ {\alpha x ^ {2}} d x
$$
Then we can calculate its first and second derivative as follows.
$$
F ^ {\prime} (\alpha) = \int_ {0} ^ {\infty} x ^ {3} r (x) e ^ {\alpha x ^ {2}} d x, \quad F ^ {\prime \prime} (\alpha) = \int_ {0} ^ {\infty} x ^ {5} r (x) e ^ {\alpha x ^ {2}} d x
$$
By the definition of  $H(\alpha)$ , we have
$$
\begin{array}{l} H (\alpha) = \int_ {0} ^ {\infty} x ^ {5} r (x) e ^ {\alpha x ^ {2}} d x - \left(a ^ {\prime 2} + b ^ {\prime 2}\right) \int_ {0} ^ {\infty} x ^ {3} r (x) e ^ {\alpha x ^ {2}} d x + a ^ {\prime 2} b ^ {\prime 2} \int_ {0} ^ {\infty} x r (x) e ^ {\alpha x ^ {2}} d x \\ = F ^ {\prime \prime} (\alpha) - (a ^ {\prime 2} + b ^ {\prime 2}) F ^ {\prime} (\alpha) + a ^ {\prime 2} b ^ {\prime 2} F (\alpha) \\ \end{array}
$$
Lemma 23 (First and Second Derivatives of  $F(0)$  when  $a' \geq 0$ ). If  $|a + b| \geq \epsilon^s$  for  $s \geq 2$  and  $a' \geq 0$ , then we have either  $F'(0) < -\frac{C_2\epsilon^{6s + 6.5}}{\log^4(1 / \epsilon)}$  or  $F''(0) > \frac{C_3\epsilon^{6s + 4.5}}{\log^4(1 / \epsilon)}$  for constants  $C_2, C_3 > 0$ .
Proof. We prove the lemma by showing that  $H(0) > \frac{C_1\epsilon^{6s + 4.5}}{\log^5(1 / \epsilon)}$ . We calculate  $H(0)$  as follows.
$$
\begin{array}{l} H (0) = \int_ {0} ^ {\infty} x \left(x ^ {2} - a ^ {\prime 2}\right) \left(x ^ {2} - b ^ {\prime 2}\right) r (x) d x \\ \geq \int_ {a ^ {\prime}} ^ {b ^ {\prime}} x (x ^ {2} - a ^ {\prime 2}) (b ^ {\prime 2} - x ^ {2}) \hat {q} (- x) d x \\ = \int_ {a ^ {\prime}} ^ {b ^ {\prime}} x \left(x ^ {2} - a ^ {\prime 2}\right) \left(b ^ {\prime 2} - x ^ {2}\right) \frac {\sigma_ {1} q \left(x \sigma_ {1} - \mu_ {1}\right)}{1 - \int_ {a} ^ {b} q (x) d x} d x \\ = \frac {\int_ {a - 2 \mu_ {1}} ^ {b - 2 \mu_ {1}} (x + \mu_ {1}) \left((x + \mu_ {1}) ^ {2} - (a - \mu_ {1}) ^ {2}\right) \left((b - \mu_ {1}) ^ {2} - (x + \mu_ {1}) ^ {2}\right) q (x) d x}{\sigma_ {1} ^ {3} \left(1 - \int_ {a} ^ {b} q (x) d x\right)} \\ = \frac {\int_ {a - 2 \mu_ {1}} ^ {b - 2 \mu_ {1}} (x + \mu_ {1}) (x + a) (x - a + 2 \mu_ {1}) (b + x) (b - x - 2 \mu_ {1}) q (x) d x}{\sigma_ {1} ^ {3} \left(1 - \int_ {a} ^ {b} q (x) d x\right)} \\ \end{array}
$$
Denote  $\rho \coloneqq \int_{a - 2\mu_1}^b q(x)dx$  , by Lemma 21 and the bound of  $b$  , we know
$$
\rho = \int_ {a - 2 \mu_ {1}} ^ {b} q (x) d x \geq \frac {\epsilon^ {s} / 2 e \cdot \epsilon}{1 + \ln (1 / \epsilon)} \geq \frac {\epsilon^ {s + 1}}{6 \ln (1 / \epsilon)}
$$
Choose  $t_1 < t_2$  such that  $\int_{a - 2\mu_1}^{t_1} q(x) \, dx = \int_{t_1}^{t_2} q(x) \, dx = \int_{t_2}^b q(x) \, dx$ . Since  $q(x)$  is upper bounded by 1 by Lemma 9, we have
$$
t _ {1} - (a - 2 \mu_ {1}) \geq \frac {\rho}{3}, b - t _ {2} \geq \frac {\rho}{3}.
$$
Using this, we can bound
$$
\begin{array}{l} \int_ {t _ {1}} ^ {t _ {2}} (x + \mu_ {1}) (x + a) (x - a + 2 \mu_ {1}) (b + x) (b - x - 2 \mu_ {1}) q (x) d x \\ \geq \left| \mu_ {1} \right| ^ {2} \frac {\rho}{3} \frac {\rho}{3} \frac {\rho}{3} \frac {\rho}{3} \geq \frac {C _ {1} \epsilon^ {6 s + 4}}{\ln^ {4} (1 / \epsilon)} \text {f o r s o m e c o n s t a n t} C _ {1} > 0 \\ \end{array}
$$
We have shown that  $\sigma_1 \leq 1 / \sqrt{1 - \int_a^b q(x) dx}$ . Combining all results, we can compute  $H(0)$  as
$$
H (0) > \sqrt {1 - \int_ {a} ^ {b} q (x) d x} \frac {C _ {1} \epsilon^ {6 s + 4}}{\log^ {4} (1 / \epsilon)} > \frac {C _ {1} \epsilon^ {6 s + 4 . 5}}{\log^ {4} (1 / \epsilon)}
$$
Given  $F(0) = 0$ , by Lemma 22,
$$
F ^ {\prime \prime} (0) = H (0) + \left(a ^ {\prime 2} + b ^ {\prime 2}\right) F ^ {\prime} (0)
$$
Since the distribution is symmetric, we know
$$
| \mu_ {1} | = \frac {\left| \int_ {| a |} ^ {b} x q (x) d x \right|}{1 - \int_ {a} ^ {b} q (x) d x} \leq \frac {\mathbb {E} _ {x \sim q} | x |}{\epsilon} \leq \frac {\sqrt {\mathbb {E} _ {x \sim q} x ^ {2}}}{\epsilon} = \frac {1}{\epsilon}
$$
So we know  $0 \leq a' < b' = b - 2\mu_1 < 2.5/\epsilon$ , thus  $(a'^2 + b'^2)^2 < 25/(2\epsilon^2)$ . Thus we have either  $F'(0) < -\frac{C_2\epsilon^{6s + 6.5}}{\log^4(1/\epsilon)}$  or  $F''(0) > \frac{C_3\epsilon^{6s + 4.5}}{\log^4(1/\epsilon)}$  for constants  $C_2, C_3 > 0$ .
Lemma 24 (Upper Bound of  $F$ 's derivatives). For  $\alpha \leq 0$ , the derivatives of  $F(\alpha)$  are bounded as
$$
| F ^ {\prime} (\alpha) | \leq C _ {4} / \epsilon^ {3}, | F ^ {\prime \prime} (\alpha) | \leq C _ {5} / \epsilon^ {5}, | F ^ {\prime \prime \prime} (\alpha) | \leq C _ {6} / \epsilon^ {7} \quad f o r c o n s t a n t s C _ {4}, C _ {5}, C _ {6} > 0.
$$
Proof. Define  $M_{k} = \mathbb{E}_{x\sim q}|x|^{k}$ . By Cauchy-Schwarz Inequality,  $M_1\leq \sqrt{M_2M_0} = 1$ . By Lemma 12,
$$
M _ {k} \leq (2 k) ^ {k} (M _ {1}) ^ {k} \leq (2 k) ^ {k}
$$
Also we have proved that  $|\mu_1| \leq 1 / \epsilon$ . By definition of  $F(\alpha)$ , we calculate its first derivative as follows.
$$
\begin{array}{l} | F ^ {\prime} (\alpha) | \leq \int_ {- \infty} ^ {\infty} q (x) | x - \mu_ {1} | ^ {3} e ^ {\alpha (x - \mu) ^ {2}} d x \\ \leq \int_ {- \infty} ^ {\infty} q (x) | x - \mu_ {1} | ^ {3} d x \\ \leq \int_ {- \infty} ^ {\infty} q (x) \left(| x | ^ {3} - 3 \mu_ {1} x ^ {2} + 3 \mu_ {1} ^ {2} | x | - \mu_ {1} ^ {3}\right) d x \\ = M _ {3} - 3 \mu_ {1} M _ {2} + 3 \mu_ {1} ^ {2} M _ {1} - \mu_ {1} ^ {3} \\ \leq C _ {4} / \epsilon^ {3} \quad \text {f o r s o m e c o n s t a n t} C _ {4} > 0 \\ \end{array}
$$
Similarly, we calculate its second and third derivatives as follows.
$$
\begin{array}{l} | F ^ {\prime \prime} (\alpha) | \leq \int_ {- \infty} ^ {\infty} q (x) | x - \mu_ {1} | ^ {5} d x \\ \leq M _ {5} - 5 \mu_ {1} M _ {4} + 1 0 \mu_ {1} ^ {2} M _ {3} - 1 0 \mu_ {1} ^ {3} M _ {2} + 5 \mu_ {1} ^ {4} M _ {1} - \mu^ {5} \\ \leq C _ {5} / \epsilon^ {5} \quad \text {f o r s o m e c o n s t a n t} C _ {5} > 0 \\ \end{array}
$$
$$
\begin{array}{l} | F ^ {\prime \prime \prime} (\alpha) | \leq \int_ {- \infty} ^ {\infty} q (x) | x - \mu_ {1} | ^ {7} d x \\ = M _ {7} - 7 \mu_ {1} M _ {6} + 2 1 \mu_ {1} ^ {2} M _ {5} - 3 5 \mu_ {1} ^ {3} M _ {4} + 3 5 \mu_ {1} ^ {4} M _ {3} - 2 1 \mu_ {1} ^ {5} M _ {2} + 7 \mu_ {1} ^ {6} M _ {1} - \mu_ {1} ^ {7} \\ \leq C _ {6} / \epsilon^ {7} \quad \text {f o r s o m e c o n s t a n t} C _ {6} > 0 \\ \end{array}
$$
Now we are ready to prove Lemma 4.
Lemma 4 (Quantitative Gap of Contrastive Mean). Suppose that  $|a + b| \geq \epsilon^5$ . Then, for  $\alpha_1 = -c_1\epsilon^{82} / d$ ,  $\alpha_2 = -c_2\epsilon^{42} / d$ , the re-weighted mean of  $P$ , denoted as  $\mu_{\alpha_1}$  and  $\mu_{\alpha_2}$ , satisfies
$$
\forall v \perp u, \quad v ^ {\top} \mu_ {\alpha_ {1}} = v ^ {\top} \mu_ {\alpha_ {2}} = 0.
$$
Proof of Lemma 4. For any  $2 \leq k \leq d$ , for any  $\alpha$ , by symmetry of  $q$ , the contrastive mean is
$$
\underset {x \sim P} {\mathbb {E}} e ^ {\alpha \| x \| ^ {2}} x _ {k} = \underset {x \sim q} {\mathbb {E}} e ^ {\alpha x ^ {2}} x \cdot \underset {x \sim \hat {q}} {\mathbb {E}} e ^ {\alpha x ^ {2}} \cdot \left(\underset {x \sim q} {\mathbb {E}} e ^ {\alpha x ^ {2}}\right) ^ {d - 2} = 0
$$
Next we will consider  $\mathbb{E}_{x\sim P}e^{\alpha \| x\|^2}x_1$ . For any  $x\geq 0$ , we have
$$
\hat {q} (x) = \frac {\sigma_ {1} q (x \sigma_ {1} + \mu_ {1})}{1 - \int_ {a} ^ {b} q (x) d x} \geq \frac {\sigma_ {1} q (- x \sigma_ {1} + \mu_ {1})}{1 - \int_ {a} ^ {b} q (x) d x} = \hat {q} (- x)
$$
$$
\underset {x \sim P} {\mathbb {E}} e ^ {\alpha \| x \| ^ {2}} x _ {1} = \underset {x _ {1} \sim \hat {q}} {\mathbb {E}} e ^ {\alpha x _ {1} ^ {2}} x _ {1} \cdot \prod_ {i = 2} ^ {d} \underset {x _ {i} \sim q} {\mathbb {E}} e ^ {\alpha x _ {i} ^ {2}} = F (\alpha) \left(\underset {x \sim q} {\mathbb {E}} e ^ {\alpha x ^ {2}}\right) ^ {d - 1}
$$
We will consider two cases depending on whether  $a' \geq 0$ . See Figure 4.1.
Firstly, if  $a' \leq 0$ . We use  $\alpha_{2}$  in this case. By Lemma 20,  $F'(0) > \epsilon^{18.5}/2$ . By Taylor expansion, there exists  $\alpha_{2} < \eta_{0} < 0$  such that
$$
F \left(\alpha_ {2}\right) = F (0) + \alpha_ {2} F ^ {\prime} (0) + \frac {\alpha_ {2} ^ {2}}{2} F ^ {\prime \prime} (\eta_ {0}).
$$
By Lemma 24, we know  $F''(\eta_0) \leq C_5 / \epsilon^5$ . Since  $F(0) = 0$ , we know for  $\alpha_{2} = -c_{2}\epsilon^{42} / d$