SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Then we rearrange  $F(\alpha)$  by combining  $e^{\alpha x^2}$  as in Figure 4.1.
If  $a' \leq 0$ , we rewrite  $F(\alpha)$  as
$$
F (\alpha) = - \int_ {- a ^ {\prime}} ^ {b ^ {\prime}} x \hat {q} (- x) e ^ {\alpha x ^ {2}} d x + \int_ {b ^ {\prime}} ^ {\infty} x (\hat {q} (x) - \hat {q} (- x)) e ^ {\alpha x ^ {2}} d x
$$
We treat  $F(\alpha)$  as the integral of  $a(x)e^{\alpha x^2}$  for  $x \geq -a'$ . Since  $\hat{q}(x) - \hat{q}(-x) > 0$  for  $x > b'$ , we have  $a(x) > 0$  for  $x \in [-a', b')$  and  $a(x) < 0$  for  $x > b'$ . In other words, for increasing  $x$ , the sign of  $a(x)$  only changes once. By Theorem 3,  $F(\alpha) = 0$  has at most one root.
If  $a^\prime >0$  , we arrange  $F(\alpha)$  in the same way and get
$$
F (\alpha) = \int_ {0} ^ {a ^ {\prime}} x (\hat {q} (x) - \hat {q} (- x)) e ^ {\alpha x ^ {2}} d x - \int_ {a ^ {\prime}} ^ {b ^ {\prime}} x \hat {q} (- x) e ^ {\alpha x ^ {2}} d x + \int_ {b ^ {\prime}} ^ {\infty} x (\hat {q} (x) - \hat {q} (- x)) e ^ {\alpha x ^ {2}} d x
$$
Similarly, we treat  $F(\alpha)$  as the integral of  $a(x)e^{\alpha x^2}$  for  $x \geq 0$ . For increasing  $x$ , the sign of  $a(x)$  changes twice. By Descartes' rule of signs,  $F(\alpha) = 0$  has at most two roots. In addition, we know  $F(0) = \mathbb{E}_P x_1 = 0$  by definition of  $P$ , which implies that  $\alpha = 0$  is one root of  $F(\alpha)$ . So there is at most one nonzero root of  $F(\alpha)$ . In other words, for any two distinct nonzero  $\alpha_1, \alpha_2$ , at least one of  $F(\alpha_1), F(\alpha_2)$  is nonzero. Consequently,
$$
\max  \left(\left| \underset {x \sim P} {\mathbb {E}} e ^ {\alpha_ {1} \| x \| ^ {2}} x _ {1} \right|, \left| \underset {x \sim P} {\mathbb {E}} e ^ {\alpha_ {2} \| x \| ^ {2}} x _ {1} \right|\right) > 0.
$$
Proof Idea of Lemma 4 (Quantitative Gap of Contrastive Mean). To get a quantitative bound on the contrastive mean, we follow the same idea of bounding the number of roots of  $F(\alpha)$  as in Lemma 1. By taking the derivative of  $F(\alpha)$ , we can show that either  $F'(0) \neq 0$  or  $F''(0) \neq 0$ . Then by Taylor expansion, we can choose two distinct  $\alpha$ 's (near zero) so that one of the corresponding contrastive means is bounded away from zero.
Moment Ratio and Contrastive Covariance. To prove Lemma 2, we develop a new monotonicity property of the moment ratio of logconcave distributions. Moment ratio is specifically defined as the ratio of the fourth moment to the square of the second moment of truncated versions of the distribution. This measurement essentially reflects the uncentered kurtosis of the distribution. We will prove the monotonicity of the moment ratio by reducing the case of general logconcave distributions to exponential distributions.
Proof of Lemma 3 (Moment Ratio). We show the monotonicity of moment ratio by showing its derivative with respect to  $t$  is negative. Define  $M_{k}(t) = \int_{t}^{\infty}x^{k}q(x)dx$ . By calculation, the derivative of moment ratio is proportional to  $-H(t)$ , where  $H(t)$  is defined as follows:
$$
H (t) = t ^ {4} M _ {0} (t) M _ {2} (t) + M _ {2} (t) M _ {4} (t) - 2 t ^ {2} M _ {0} (t) M _ {4} (t).
$$
Let  $h(x) = \beta e^{-\gamma x}$  be an exponential function  $(\beta, \gamma > 0)$  such that
$$
M _ {0} (t) = N _ {0} (t), M _ {2} (t) = N _ {2} (t), \text {w h e r e} N _ {k} (t) = \int_ {t} ^ {\infty} x ^ {k} h (x) d x, k \in \mathbb {N}.
$$
$$
\int_ {t} ^ {\infty} (h (x) - q (x)) d x = 0, \int_ {t} ^ {\infty} x ^ {2} (h (x) - q (x)) = 0
$$
By the logconcavity of  $q$ , the graph of  $h$  intersects with the graph of  $q$  at exactly two points  $u' < v$  where  $v > 0$ . Also we have  $h(x) \leq q(x)$  at the interval  $[u', v]$  and  $h(x) > q(x)$  outside the interval. Let  $u = \max \{0, u'\}$ . So for  $x \geq 0$ ,  $(x - u)(x - v)$  has the same sign as  $h(x) - q(x)$ . Since  $t \geq 0$ , we have
$$
\int_ {t} ^ {\infty} (x ^ {2} - u ^ {2}) (x ^ {2} - v ^ {2}) (h (x) - q (x)) \geq 0
$$
Expanding the terms and we get
$$
\int_ {t} ^ {\infty} x ^ {4} (h (x) - q (x)) \geq (u ^ {2} + v ^ {2}) \int_ {t} ^ {\infty} x ^ {2} (h (x) - q (x)) d x - u ^ {2} v ^ {2} \int_ {t} ^ {\infty} (h (x) - q (x)) d x = 0
$$
This shows that  $N_{4}(t) \geq M_{4}(t)$ . To show that  $H(t) > 0$ , we consider two cases.
Firstly if  $M_2(t) - 2t^2 \geq 0$ , we have
$$
H (t) = t ^ {4} M _ {0} (t) M _ {2} (t) + M _ {4} (t) \left(M _ {2} (t) - 2 t ^ {2}\right) > 0.
$$
Secondly if  $M_2(t) - 2t^2 < 0$ , by calculation of the exponential function's moments, we have
$$
t ^ {4} N _ {0} (t) N _ {2} (t) > - N _ {4} (t) \left(N _ {2} (t) - 2 t ^ {2}\right)
$$
$$
H (t) = t ^ {4} M _ {0} (t) M _ {2} (t) + M _ {4} (t) \left(M _ {2} (t) - 2 t ^ {2}\right) \geq \left(M _ {2} (t) - 2 t ^ {2}\right) \left(M _ {4} (t) - N _ {4} (t)\right) \geq 0
$$
The equality holds if and only if  $M_4(t) = N_4(t) > 0$ . Then we have
$$
H (t) = t ^ {4} N _ {0} (t) N _ {2} (t) + N _ {4} (t) (N _ {2} (t) - 2 t ^ {2}) > 0.
$$
Combining both cases,  $\mathrm{mr}_q'(t) < 0, \forall t \geq 0$ , which implies that the moment ratio of  $q$  is strictly decreasing with respect to  $t$ .
Proof Idea of Lemma 2 (Constractive Covariance). View the spectral gap of the re-weighted covariance, denoted as  $\lambda_1(\tilde{\Sigma}) - \lambda_2(\tilde{\Sigma})$ , as a function  $S(\alpha)$ . By calculation,  $S(0) = 0$  and  $S'(0)$  is proportional to  $\mathrm{mr}_q(b) - \mathrm{mr}_q(0)$ , which is negative by the monotonicity property of moment ratio. Then we can prove the spectral gap for the chosen  $\alpha$  using Taylor expansion.
Proof Idea of Lemma 5 (Quantitative Gap of Contrastive Covariance). We first prove the case when  $a = -b$ , and then extend the result to the near-symmetric case when  $|a + b| < \epsilon^5$  by comparing the re-weighted second moments of two distributions created by restricting  $q$  to  $\mathbb{R} \backslash [a, b]$  and  $\mathbb{R} \backslash [-b, b]$  respectively.
The following theorem enables us to bound the sample complexity to estimate the covariance matrix.
Lemma 6 (Covariance Estimation [SV13]). Consider independent isotropic random vectors  $X_{i}$  in  $\mathbb{R}^d$  s.t. for some  $C,\eta >0$  , for every orthogonal projection  $P$  in  $\mathbb{R}^d$
$$
\mathbb {P} \left(\| P X _ {i} \| > t\right) \leq C t ^ {- 1 - \eta} f o r t > C r a n k (P).
$$
Let  $\epsilon \in (0,1)$ . Then with the sample size  $N = O(d\epsilon^{-2 - 2 / \eta})$ , we have
$$
\mathbb {E} \| \Sigma - \hat {\Sigma} \| _ {o p} \leq \epsilon \| \Sigma \| _ {o p}.
$$
The following classical theorem will allow us to use the eigenvalue gap to identify the relevant vector.
Lemma 7 (Davis-Kahan [DK70]). Let  $S$  and  $T$  be symmetric matrices with the same dimensions. For a fixed  $i$ , assume that the largest eigenvalue of  $S$  is well separated from the second largest eigenvalue of  $S$ , i.e.  $\exists \delta > 0$  s.t.  $\lambda_1(S) - \lambda_2(S) > \delta$ . Then for the top eigenvectors of  $S$  and  $T$ , we have
$$
\sin \theta (v _ {1} (S), v _ {1} (T)) \leq \frac {2 \| S - T \| _ {o p}}{\delta}.
$$
Proof Sketch of Main Theorem. We prove the theorem by considering whether the removed band  $[a, b]$  is symmetric or not. If  $|a + b| \geq \epsilon^5$ , by Lemma 4, for the chosen two  $\alpha$ s, at least one of the contrastive means  $\tilde{\mu}_i$  along  $u$  direction is bounded away from zero (by  $\mathrm{poly}(1 / d, \epsilon)$ ) and the projection of the contrastive mean along any direction orthogonal to  $u$  is zero by symmetry. So by Chebyshev's Inequality, we will ensure that the angle between  $u$  and  $\tilde{\mu}_i$  is less than  $\delta$  with high probability using  $O(\mathrm{poly}(d, 1 / \epsilon, 1 / \delta))$  samples. On the other hand, if  $|a + b| < \epsilon^5$ , we rely on the contrastive covariance. By Lemma 5, the top eigenvector  $v$  aligns with  $u$  while the top two eigenvalues  $\lambda_1 > \lambda_2$  satisfy  $\lambda_1 - \lambda_2 > \mathrm{poly}(\epsilon)$ . By Lemma 7, we can upper bound the angle between  $u$  and  $v$  by the quotient of the operator norm of the difference of the contrastive covariance and sample contrastive covariance and the spectral gap  $\lambda_1 - \lambda_2$ . The covariance matrix itself can be estimated to desired accuracy efficiently with  $O(\mathrm{poly}(d, 1 / \epsilon, 1 / \delta))$  samples by Lemma 6. This ensures the closeness of  $u$  and top eigenvector  $v$  and hence a TV-distance guarantee. The formal statement of the theorem and the proofs are included in the Appendix.
While our primary goal is to establish polynomial bounds on the sample and time complexity, our algorithms are natural and easy to implement. We study the efficiency and performance of Algorithm 1 on data drawn from affine product distributions with margin. Here we consider three special cases of logconcave distribution: Gaussian, uniform in an interval and exponential. We include four experiments. In all results, we measure the performance of the algorithm using the sin of the angle between the true normal vector  $u$  and the predicted vector  $\hat{u}$ , i.e.,  $\sin \theta(u, \hat{u})$ , which bounds the  $TV$  distance between the underlying distribution and the predicted one after isotropic transformation. Experimental results strongly suggest that the sample complexity is a small polynomial, perhaps even just nearly linear in both the dimension and the separation parameter  $\epsilon$ .
Overall Performance. Here we conduct the experiments based on a grid search of  $(a,b)$  pairs on three special cases of logconcave distribution: Gaussian, uniform in an interval and exponential. We measure the performance of  $(a,b)$  pairs, where for each pair of  $(a,b)$ , we conduct five independent trials. For Gaussian and exponential distribution, we choose  $-3 \leq a < b \leq 3$  and for uniform distribution, we choose  $-1.5 \leq a < b \leq 1.5$ . Here we set the dimension  $d = 10$  and sample size  $N = 1000000$ . For the parameters, we choose  $\alpha_{1} = \alpha_{3} = -0.1$ ,  $\alpha_{2} = -0.2$ . See Figure 5.1 as the heatmap of  $\sin \theta(u,\hat{u})$  given different pairs of  $(a,b)$ .
Although in Algorithm 1, we use extremely small values of the weight parameter  $\alpha$ , our experiments show that larger constant values also work empirically, leading to much smaller sample complexity. This coincides with our qualitative lemmas (Lemma 1, Lemma 2).
The algorithm performs well as seen in the results, except when  $a$  and  $b$  are both close to the edge, and thus there is almost no mass on one side of the band. Also, the uniform distribution is the easiest to learn, while the exponential is the hardest among these three distributions. As shown in all three plots, the algorithm performs the best when  $a$  and  $b$  are near symmetric with origin. In other words, contrastive covariance has better sample complexity than contrastive mean when we fix other hyperparameters. This coincides with our sample complexity bounds as in the proof of Theorem 1.
(b) Uniform
(c) Exponential
Figure 5.1: The performance of Algorithm 1 with varying  $(a,b)$
Performance of Contrastive Mean and Covariance. In this experiment, we fix a negative  $a$  as the left endpoint of the removed band, and measure the performance of both contrastive mean and contrastive covariance with respect to different margin right endpoint  $b$ . As shown in Figure 5.2, contrastive mean performs well except when  $a + b$  is close to zero, while contrastive covariance performs well only when  $a + b$  is close to zero. This coincides with our algorithm and analysis for the two cases. In addition, our algorithm chooses the best normal vector among candidates from both contrastive mean and covariance. So our algorithm achieves good performance (minimum of contrastive mean and covariance curves).
Specifically, we choose  $a = -2, b \in [-1.9, 4]$  for Gaussian and exponential cases, and  $a = -0.5, b \in [-0.4, 0.9]$  for uniform case. We choose the dimension  $d = 10$ , the sample size  $N = 2000000$ . We choose  $\alpha_{1} = \alpha_{3} = -0.1, \alpha_{2} = -0.2$ . We average the result with 50 independent trials.
(a) Gaussian,  $a = -2$
Figure 5.2: The performance of Algorithm 1 for fixed  $a$  and varying  $b$ . The yellow lines show the result computed using the top eigenvector of the contrastive covariance. The blue dotted lines show the better of the two contrastive means.
(b) Uniform,  $a = -0.5$
(c) Exponential,  $a = -2$
Dimension Dependence. In this experiment, we show the relationship between the input dimension  $d$  and the sample complexity. For fixed number size  $N = 1000000$ , we measure the performance of our algorithm with different  $d$ . The result is averaged based on a grid search of  $(a, b)$  pairs, where for each pair of  $(a, b)$ , we conduct five independent trials. For Gaussian and exponential distributions, we choose  $-3 \leq a < b \leq 3$  and for uniform distribution, we choose  $-0.8 \leq a < b \leq 0.8$ .
As shown in Figure 5.3, the performance scales linearly with growing dimension  $d$ , suggesting a linear relationship between the sample complexity and the input dimension.
(b) Uniform
Figure 5.3: The performance of Algorithm 1 for fixed sample size  $N$  and varying dimension  $d$ .
(c) Exponential
$\epsilon$ -Dependence. To further understand the dependence on the separation parameter  $\epsilon$ , we plot the performance versus  $1 / \epsilon$  in Figure 5.4. Here we calculate  $1 / \epsilon$  as  $1 / q([a,b])$ , and the performance as the median  $\sin \theta(u,\hat{u})$  for specific mass  $q([a,b])$ . As we can see the performance drops near linearly with respect to  $1 / \epsilon$ , which indicates that the sample complexity is possibly linear in  $1 / \epsilon$  as well.
(a) Gaussian
Figure 5.4: The performance with respect to  $1 / \epsilon$ .
We have proposed and analyzed an efficient algorithm to learn the symmetric product logconcave distribution without labeled data. This is also connected to self-supervised learning. Specifically, contrastive learning without data augmentation is closely related to the contrastive covariance part in our algorithm.
The algorithm delivers more than the theoretical analysis. While our analysis focuses on specific values of  $\alpha$ , as demonstrated by the qualitative lemmas (Lemma 1, Lemma 2), any distinct pair of nonzero  $\alpha$  values should work for contrastive mean, and any bounded small  $\alpha$  should work for contrastive covariance. This flexibility ensures the applicability of the algorithm in various real-world scenarios. Furthermore, our experimental results align with this claim.
The experiments reveals a linear relationship with the input dimension  $d$ , raising an open question regarding the improvement of the sample complexity bound to be linear with respect to  $d$  (as well as  $1 / \epsilon$ ). Additionally, it might be possible to extend the algorithm's application to more general distributions.
Acknowledgements. This work was supported in part by NSF awards CCF-2007443 and CCF-2134105 and an ARC fellowship.
[SBS+00] Alexander J Smola, Peter J Bartlett, Dale Schuurmans, Bernhard Scholkopf, et al. Advances in large margin classifiers. MIT press, 2000.
[SV13] Nikhil Srivastava and Roman Vershynin. Covariance estimation for distributions with  $2 + \epsilon$  moments. The Annals of Probability, 41(5):3081-3111, 2013.
[TV18] Yan Shuo Tan and Roman Vershynin. Polynomial time and sample complexity for nongaussian component analysis: Spectral methods. In Conference On Learning Theory, pages 498-534. PMLR, 2018.
[Val98] Leslie G Valiant. Projection learning. In Proceedings of the eleventh annual conference on Computational learning theory, pages 287-293, 1998.
[Vem10] Santosh S Vempala. Learning convex concepts from gaussian distributions with pca. In 2010 IEEE 51st Annual Symposium on Foundations of Computer Science, pages 124-130. IEEE, 2010.
[VX11] Santosh S Vempala and Ying Xiao. Structure from local optima: Learning subspace juntas via higher order pca. arXiv preprint arXiv:1108.3329, 2011.
[XYNLS04] Linli Xu, James Neufeld, Bryce Larson, and Dale Schuurmans. Maximum margin clustering. Advances in neural information processing systems, 17, 2004.
A.1 Logconcave Distributions
Lemma 8 (Lemma 5.4, [LV07]). Let  $X$  be a random point drawn from a one-dimensional logconcave distribution. Then
$$
\mathbb {P} (X \geq \mathbb {E} X) \geq \frac {1}{e}.
$$
Lemma 9 (Lemma 5.5,[LV07]). Let  $p: \mathbb{R} \to \mathbb{R}_+$  be an isotropic logconcave density function. Then we have
(a) For all  $x,g(x)\leq 1$
(b)  $g(0)\geq 1 / 8$
Lemma 10 (Lemma 5.6,[LV07]). Let  $X$  be a random point drawn from a logconcave density function  $p: \mathbb{R} \to \mathbb{R}_+$ . Then for every  $c \geq 0$ ,
$$
\mathbb {P} (p (X) \leq c) \leq \frac {c}{\max  _ {x} p (x)}
$$
Lemma 11 (Lemma 5.7, [LV07]). Let  $X$  be a random variable drawn from a logconcave distribution in  $\mathbb{R}$ . Assume that  $\mathbb{E}X^2 \leq 1$ . Then for any  $t \geq 1$ , we have
$$
\mathbb {P} (X \geq t) \leq e ^ {1 - t}
$$
Corollary 1. Let  $X$  be a random point drawn from an isotropic symmetric logconcave density function  $p: \mathbb{R} \to \mathbb{R}_+$ . Then we have for  $t \geq 0$ , we have
$$
\mathbb {P} (X \geq t) \leq 8 p (t)
$$
Proof. Since  $p(x)$  is symmetric, we know  $p(x)$  is monotonically decreasing for  $x \geq 0$ . Then we apply Lemma 10 with  $c = p(x)$ , and get
$$
\mathbb {P} (x \geq t) \leq \mathbb {P} (p (X) \leq p (t)) \leq \frac {p (t)}{\max _ {x} p (x)}
$$
On other hand, by Lemma 9, we have  $\max_x p(x) \geq p(0) \geq 1/8$ . So we have
$$
\mathbb {P} (X \geq t) \leq 8 p (t).
$$
Lemma 12 (Theorem 5.22, [LV07]). For a random point  $X$  drawn from a logconcave distribution in  $\mathbb{R}^d$ , then
$$
\mathbb {E} | X | ^ {k} \leq (2 k) ^ {k} (\mathbb {E} (| X |)) ^ {k}
$$
Lemma 13. Let  $X$  be a random point drawn from an isotropic symmetric logconcave density function  $p: \mathbb{R} \to \mathbb{R}_+$ . Then for any  $t \geq 3$ , we have
$$
p (t) \leq p (0) \cdot 2 ^ {- t / 3}
$$
Proof. First we claim that  $p(3) < p(0) / 2$ . Otherwise,
$$
\mathbb {E} X ^ {2} \geq \int_ {0} ^ {3} x ^ {2} p (x) d x \geq \frac {p (0)}{2} \frac {3 ^ {3}}{3} \geq \frac {9}{1 6} > \frac {1}{2}
$$
This leads to the contradiction. Then for any  $t \geq 3$ , from the logconcavity of  $p$
$$
p (3) \geq p (0) ^ {1 - \frac {3}{t}} p (t) ^ {\frac {3}{t}}
$$
A.2 Descartes' Rule of Signs
Descartes' rule of signs is a well-known principle in algebra that offers a way to estimate the maximum number of real roots for any polynomial. This classical theorem can be stated as follows:
Theorem 4 (Descartes' Rule of Signs). For the generalized Dirichlet polynomial
$$
F (x) = \sum_ {j = 1} ^ {n} a _ {j} e ^ {p _ {j} x}, p _ {1} \geq p _ {2} \geq \dots \geq p _ {n},
$$
the number of roots of  $F(x) = 0$  is at most the number of sign changes in the series  $\{a_1, a_2, \dots, a_n\}$ .
In this section, we state and prove a variant of Descartes' rule of signs in the integral form, which we apply directly to prove Lemma 1. To begin with, we say a function  $f$  has a root of order  $k$  at point  $x$  if
$$
f (x) = f ^ {\prime} (x) = \dots = f ^ {(k - 1)} (x) = 0 \text {a n d} f ^ {(k)} (x) \neq 0.
$$
We denote  $Z(f)$  as the number of roots of  $f$ , counted with their orders. Then we can show that the number of roots of  $f$  is upper bounded by one plus the number of roots of  $f'$  in Lemma 14. We use Rolle's Theorem in the proof of the lemma.
Theorem 5 (Rolle's Theorem). Suppose that a function  $f$  is differentiable at all points of interval  $[a, b]$  and  $f(a) = f(b)$ . Then there is at least one point  $x_0 \in (a, b)$  such that  $f'(x_0) = 0$ .
Lemma 14.  $Z(f) \leq Z(f') + 1$ .
Proof. Suppose  $f$  has a root of order  $k_r$  as  $x_r, 1 \leq r \leq n$ . Then  $f'$  has a root of order  $k - 1$  at  $x_r$ . These add up to
$$
\sum_ {r = 1} ^ {n} \left(k _ {r} - 1\right) = Z (f) - n
$$
By Rolle's Theorem,  $f'(x)$  also has at least  $n - 1$  roots in the gaps between the points  $x_r$ . Together, these two facts give
$$
Z (f ^ {\prime}) \geq Z (f) - n + n - 1 = Z (f) - 1.
$$
Theorem 3 (Descartes' Rule of Signs in the Integral Form). Let  $F(\alpha) = \int_0^\infty a(x)e^{\alpha x^2}dx$ . Then the number of roots of  $F(\alpha) = 0$  is at most the number of sign changes in  $a(x)$  for  $x\geq 0$ .
Proof. We prove the theorem with induction on the number of sign changes of  $a(x)$ . For the base case when the number of sign changes of  $a(x)$  is zero, we assume wlog that  $a(x) \geq 0$ . Then  $F(\alpha) > 0$  and thus  $F(\alpha)$  has no root. Now we assume that the theorem holds when the number of sign changes is  $t$  and we will show the  $t + 1$  case.
Let one of the sign changes of  $a(x)$  occurs at  $x_0$ . Define
$$
F _ {0} (\alpha) := \int_ {0} ^ {\infty} a (x) e ^ {\alpha \left(x ^ {2} - x _ {0} ^ {2}\right)} d x,
$$
which has the same roots as  $F(\alpha)$ . By taking derivative, we get
$$
F _ {0} ^ {\prime} (\alpha) = \int_ {0} ^ {\infty} a (x) \left(x ^ {2} - x _ {0} ^ {2}\right) e ^ {\alpha \left(x ^ {2} - x _ {0} ^ {2}\right)} d x.
$$
Let  $b(x) = a(x)(x^2 - x_0^2)$  be the new sequence. Then  $b(x)$  has one less sign changes than  $a(x)$ . By induction hypothesis, the number of roots of  $F_0'(\alpha)$  is upper bounded by the number of sign changes of  $b(x)$ . By Lemma 14, the number of roots of  $F_0$  is upper bounded by the number of sign changes of  $a(x)$ , thus leading to the induction step.
B Warm-up: Isotropic Isoperimetric Distribution with  $\epsilon$ -Margin
As a warm-up, we consider the isotropic product distributions with  $\epsilon$ -margin. Notably, without applying an unknown transformation on data, we can extend the logconcave distributions to isoperimetric distributions. In this section, we will demonstrate how to retrieve the normal vector  $u$  by calculating the empirical mean and top eigenvector of the empirical uncentered covariance matrix. This technique is similar to Principal Component Analysis (PCA), but instead of computing covariance matrix, we use the uncentered covariance matrix.
Definition 2. A distribution  $p$  with support  $\mathbb{R}$  is  $\psi$ -isoperimetric if there exists  $\psi > 0$  such that for any  $x \in \mathbb{R}$ , we have  $p(x) \geq \psi \min \{p([x, \infty)), p(-\infty, x]\}$ .
Definition 3 (Isotropic Isoperimetric Distribution with  $\epsilon$ -Margin). Let  $q_{1},\ldots ,q_{d}$  be symmetric one-dimensional isotropic  $\psi$ -isoperimetric density functions bounded by  $\tau$ . Let  $Q = q_{1}\otimes \dots \otimes q_{d}$ . Let  $\hat{q}$  be the density obtained after restricting  $q_{1}$  to  $\mathbb{R}\backslash [a,b]$  where  $q_{1}((- \infty ,a])\geq \epsilon ,q_{1}([a,b])\geq \epsilon$  and  $q_{1}([b,\infty))\geq \epsilon$ . Let  $P$  be an arbitrary rotation of  $\hat{q}\otimes q_2\otimes \dots \otimes q_d$ . We refer  $P$  as an Isotropic Isoperimetric Distribution with  $\epsilon$ -Margin. Let  $u$  be the unit vector normal to the margin.
Problem. Given input parameters  $\epsilon, \delta > 0$  and access to iid samples from  $P$ , an isotropic isoperimetric distribution with  $\epsilon$ -margin, the learning problem is to compute a unit vector  $\tilde{u}$  that approximates  $u$  to within TV distance  $\delta$ . That is, the TV distance between the corresponding  $\tilde{P}$  and  $P$  is at most  $\delta$ , where  $\tilde{P}$  is the distribution with margin normal to  $\tilde{u}$ .
B.1 Algorithm
Given data drawn from  $P$ , we compute the sample mean and the top eigenvector of the uncentered covariance matrix. Then we compare the max margin along these two candidate normal vectors. This gives an efficient algorithm for the problem with no re-weighting. We state the algorithm formally in Algorithm 2.
Algorithm 2 Unsupervised Halfspace Learning from Isotropic Isoperimetric Data
Input: Unlabeled data  $\{x^{(1)},\dots ,x^{(N)}\} \subset \mathbb{R}^d$ $\epsilon ,\delta >0$
- Compute the sample mean and uncentered covariance matrix:
$$
\hat {\mu} = \frac {1}{N} \sum_ {j = 1} ^ {N} x ^ {(j)}, \hat {\Sigma} = \frac {1}{N} \sum_ {j = 1} ^ {N} x ^ {(j)} x ^ {(j) ^ {\top}}
$$