SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Xinyuan Cao
xcao78@gatech.edu
Santosh S. Vempala
vempala@gatech.edu
We give a polynomial-time algorithm for learning high-dimensional halfspaces with margins in  $d$ -dimensional space to within desired Total Variation (TV) distance when the ambient distribution is an unknown affine transformation of the  $d$ -fold product of an (unknown) symmetric one-dimensional logconcave distribution, and the halfspace is introduced by deleting at least an  $\epsilon$  fraction of the data in one of the component distributions. Notably, our algorithm does not need labels and establishes the unique (and efficient) identifiability of the hidden halfspace under this distributional assumption. The sample and time complexity of the algorithm are polynomial in the dimension and  $1/\epsilon$ . The algorithm uses only the first two moments of suitable re-weightings of the empirical distribution, which we call contrastive moments; its analysis uses classical facts about generalized Dirichlet polynomials and relies crucially on a new monotonicity property of the moment ratio of truncations of logconcave distributions. Such algorithms, based only on first and second moments were suggested in earlier work, but hitherto eluded rigorous guarantees.
Prior work addressed the special case when the underlying distribution is Gaussian via Non-Gaussian Component Analysis. We improve on this by providing polytime guarantees based on TV distance, in place of existing moment-bound guarantees that can be super-polynomial. Our work is also the first to go beyond Gaussians in this setting.
Suppose points in  $\mathbb{R}^d$  are labeled according to a linear threshold function (a halfspace). Learning a threshold function from labeled examples is the archetypal well-solved problem in learning theory, in both the PAC and mistake-bound models; its study has led to efficient algorithms, a range of powerful techniques and many interesting learning paradigms. While the sample complexity in general grows with the dimension, when the halfspace has a margin, the complexity can instead be bounded in terms of the reciprocal of the squared margin width [PCST99, SBS $^{+}$ 00, AV06, LS11]. The problem is also very interesting for special classes of distributions, e.g., when the underlying distribution is logconcave, agnostic learning is possible [KKMS08], and active learning needs fewer samples compared to the general case [BBL06].
The main motivation for our work is learning a halfspace with a margin with no labels, i.e., unsupervised learning of halfspaces. This is, of course, impossible in general — there could be multiple halfspaces with margins consistent with the data — raising the question: Can there be natural distributional assumptions that allow the unsupervised learning of halfspaces? For example, suppose data is drawn from a Gaussian in  $\mathbb{R}^d$  with points in an unknown band removed, i.e., we assume there exists a unit vector  $u \in \mathbb{R}^d$  and an interval  $[a, b]$  so that the input distribution is the Gaussian restricted to the set  $\{x \in \mathbb{R}^d | \langle u, x \rangle \leq a$  or  $\langle u, x \rangle \geq b\}$ . Can the vector  $u$  be efficiently learned? Such a distributional assumption ensures that the band normal to  $u$  is essentially unique, leaving open the question of whether it can be efficiently learned.
Such models have been considered in the literature, notably for Non-Gaussian Component Analysis (NGCA)  $\left[\mathrm{BKS}^{+}06\right.$ , TV18], learning relevant subspaces [Blu94, VX11] and low-dimensional convex concepts [Vem10] where data comes from a product distribution with all components being Gaussian except for one (or a small number). It is assumed that the non-Gaussian component differs from Gaussian in some low moment and the goal is to identify this component. Another related model is Independent Component Analysis (ICA) where the input consists of samples from an affine transformation of a product distribution and the goal is to identify the transformation itself [Com94, Car98, GVX14, JKV23]. For this problem to be well-defined, it is important that at most one component of the product distribution is Gaussian. No such assumption is needed for NGCA or the more general problem we consider here.
Formally, we consider the following model and problem, illustrated in Figure 1.1.
Definition 1 (Affine Product Distribution with  $\epsilon$ -Margin). Let  $q$  be a symmetric one-dimensional isotropic logconcave density function. Let  $Q$  be the  $d$ -fold product distribution obtained from  $q$ . Let  $\hat{q}$  be the isotropized density obtained after restricting  $q$  to  $\mathbb{R} \backslash [a, b]$  where  $q((- \infty, a]) \geq \epsilon$ ,  $q([a, b]) \geq \epsilon$  and  $q([b, \infty)) \geq \epsilon$ . Let  $P$  be the product of one copy of  $\hat{q}$  and  $d - 1$  copies of  $q$ . Let  $\widehat{P}$  be obtained by a full-rank affine transformation of  $P$ ; we refer to  $\widehat{P}$  as an Affine Product Distribution with  $\epsilon$ -Margin. Let  $u$  be the unit vector normal to the margin before transformation.
(a) Symmetric and asymmetric margin  $[a,b]$ . We use contrastive covariance and mean respectively.
(b) Definition 1 with uniform  $q$ .  $\widehat{P}$  is a full-rank affine transformation of the product distribution  $P$ .
(c) Unlabeled data is drawn from  $\widehat{P}$ . The goal is to learn the normal vector  $u$ .
Figure 1.1: Affine Product Distribution with Margin.
With this model in hand, we have the following algorithmic problem.
Problem. Given input parameters  $\epsilon, \delta > 0$  and access to iid samples from  $\widehat{P}$ , an affine product distribution with  $\epsilon$ -margin, the learning problem is to compute a unit vector  $\tilde{u}$  that approximates  $u$  to within  $TV$  distance  $\delta$ . That is, the TV distance between the corresponding  $\tilde{P}$  and  $P$  is at most  $\delta$ , where  $\tilde{P}$  is the distribution with margin normal to  $\tilde{u}$ .
In this formulation of the problem with a TV distance guarantee, if each side of the halfspace receives a different label, then the probability that the output halfspace of the data disagrees with the true label (up to swapping the labels) is at most  $\delta$ .
A natural approach to identifying the halfspace is maximum margin clustering [XNLS04]: find a partition of the data into two subsets s.t. the distance between the two subsets along some direction is maximized. Unfortunately, this optimization problem is NP-hard, even to approximate.
There are at least two major difficulties we have to address. The first is the unknown affine transformation, which we cannot hope to completely identify in general. The second is that, even if we reversed the transformation, the halfspace normal is in an arbitrary direction in  $\mathbb{R}^d$  and would be undetectable in almost all low-dimensional projections, i.e., we have a needle in a haystack problem.
1.1 Results and Techniques
We give an efficient algorithm for the unsupervised halfspace learning problem under any symmetric product logconcave distribution. It consists of the following three high-level steps.
(1) Make the data isotropic (mean zero and covariance matrix identity).
(2) Re-weight data and compute the re-weighted mean  $\tilde{\mu}_i$  and the top eigenvector  $v$  of the re-weighted covariance.
(3) Project data along the vectors  $\tilde{\mu}_i, v$ , and output the vector with the largest margin.
Although the algorithm is simple and intuitive, its analysis has to overcome substantial challenges. Our main result is the following. The formal version (Theorem 7) is in the Appendix.
Theorem 1 (Main). There is an algorithm that can learn any affine product distribution with  $\epsilon$ -margin to within TV distance  $\delta$  with time and sample complexity that are polynomial in  $d, 1/\epsilon$  and  $1/\delta$  with high probability.
To see the idea of the algorithm, we first consider the case when no affine transformation is applied. In this case, we can detect the direction  $u$  by calculating the empirical mean and top eigenvector of the empirical uncentered covariance matrix. If the margin  $[a,b]$  lies on one side of the origin, the mean along  $u$  is nonzero while the mean in any other direction that is orthogonal to  $u$  is zero. Thus the mean itself reveals the vector  $u$ . Otherwise, we can show that the second moment along  $u$  is higher than along any other orthogonal direction. Thus, there is a positive gap between the top two eigenvalues of the uncentered covariance matrix and the top eigenvector is  $u$ . In fact, the algorithm applies more generally, to the product distribution created from one-dimensional bounded isoperimetric distributions. A one-dimensional distribution  $p$  is isoperimetric if there exists a constant  $\psi > 0$  such that for any  $x \in \mathbb{R}$ ,  $p(x) \geq \psi \min \{p([x,\infty)), p(-\infty,x]\}$ . The theorem is stated as follows. The formal version (Theorem 6), algorithm and proofs are included in the Appendix.
Theorem 2 (Isotropic Isoperimetric Distribution). There is an algorithm that can learn any isotropic isoperimetric bounded product distribution with  $\epsilon$ -margin to within TV distance  $\delta$  with time and sample complexity that are polynomial in  $d, 1/\epsilon, 1/\delta$  with high probability.
In the general case, when an unknown affine transformation is applied, the algorithm first computes the empirical mean and covariance of the sample and makes the empirical distribution isotropic. Then we will consider two cases as illustrated in Figure 1.1a. If the unknown band is not centered around the mean along  $u$ , we can expect the empirical mean to differ from the mean of the underlying product distribution without the margin. Consequently, if we knew the latter, we can use the difference to estimate  $u$ . However, in general, we do not have this information. Instead, we demonstrate that there exists a re-weighting of the sample so that re-weighted empirical mean compared to the unweighted empirical mean is a good estimate of  $u$ . In other words, with appropriate re-weighting, the mean shifts along the normal direction to the unknown band. On the other hand, if the band is centered along  $u$ , the mean shift will be zero. In this scenario, we will show that the maximum eigenvector of a re-weighted covariance matrix is nearly parallel to  $u$ !
Our algorithm only uses first and second order moments, can be implemented efficiently, and is in fact practical (see Section 5). The main challenges are (1) proving the existence of band-revealing re-weightings and (b) showing that a polynomial-sized sample (and polynomial time) suffices.
To prove the main theorem, we will show that either the re-weighted mean induces a contrastive gap (Lemma 1), or the eigenvalues of the re-weighted covariance matrix induce a contrastive gap (Lemma 2). In the subsequent two lemmas, we adopt the notation from Definition 1. Here,  $P$  represents a product distribution with  $\epsilon$ -margin defined by the interval  $[a,b]$  (before transformation).
Lemma 1 (Contrastive Mean). If  $|a + b| > 0$ , then for any two distinct nonzero  $\alpha_{1}, \alpha_{2} \in \mathbb{R}$ , at least one of the corresponding re-weighted means is nonzero, i.e.,
$$
\max  \left(\left| \underset {x \sim P} {\mathbb {E}} e ^ {\alpha_ {1} \| x \| ^ {2}} u ^ {\top} x \right|, \left| \underset {x \sim P} {\mathbb {E}} e ^ {\alpha_ {2} \| x \| ^ {2}} u ^ {\top} x \right|\right) > 0.
$$
Lemma 2 (Contrastive Covariance). If  $a + b = 0$ , then there exists an  $\alpha < 0$ , such that (1) there is a positive gap between the top two eigenvalues of the re-weighted uncentered covariance matrix  $\tilde{\Sigma} = \mathbb{E}_{x \sim P} e^{\alpha \|x\|^2} (xx^\top)$ . That is,  $\lambda_1(\tilde{\Sigma}) > \lambda_2(\tilde{\Sigma})$ . (2) The top eigenvector of  $\tilde{\Sigma}$  is  $u$ .
The proof of Lemma 1 uses Descartes' rule of signs applied to a suitable potential function. To prove Lemma 2, we develop a new monotonicity property of the moment ratio (defined as the ratio of the variance of  $X^2$  and the squared mean of  $X^2$ ) for truncations of logconcave distributions. The moment ratio is essentially the square of the coefficient of variation of  $X^2$ . An insight from the monotonicity of the moment ratio is that for logconcave distributions with positive support, when the distribution is restricted to an interval away from the origin, it needs a smaller sample size to estimate its second moment accurately. We state the lemma as follows.
Lemma 3 (Monotonicity of Moment Ratio). Let  $q$  be a logconcave distribution in one dimension with nonnegative support. For any  $t \geq 0$ , let  $q_t$  be the distribution obtained by restricting  $q$  to  $[t, \infty)$ . Then the moment ratio of  $q$ , defined as  $mr_q(t) := \frac{\mathrm{var}_{q_t}(X^2)}{(\mathbb{E}_{q_t} X^2)^2}$ , is strictly decreasing with  $t$ .
To obtain polynomial guarantees, we will need quantitative estimates of the inequalities in the above two lemmas. Establishing such quantitative bounds is the bulk of the technical contribution of this paper. While our focus is on proving polynomial bounds, whose existence a priori is far from clear, we did not optimize the polynomial bounds themselves; our experimental results suggest that in fact the dependence on both  $d$  and  $1 / \epsilon$  might be linear!
Efficient algorithms for supervised halfspace learning [Ros58, MP69], combined with the kernel trick  $\mathrm{[CST^{+}00, HDO^{+}98]}$ , serve as the foundation of much of learning theory. Halfspaces with margin are also well-studied, due to their motivation from the brain, attribute-efficient learning [Val98, Blu90], random projection based learning [AV06], and turn out to have sample complexity that grows inverse polynomially with the margin, independent of the ambient dimension. When examples are drawn from a unit Euclidean ball in  $\mathbb{R}^d$ , and the halfspace has margin  $\gamma$ , then the sample complexity grows as  $O(1 / \gamma^2)$  regardless of the dimension. This leads to the question of whether labels are even necessary, or the halfspace can be identified from unlabeled samples efficiently — the focus of the present paper.
The model of unsupervised learning we study is similar to other classical models in the literature, notably Independent Component Analysis where input data consists of iid samples from an unknown affine transformation of a product distribution. There, the goal is to recover the affine transformation under minimal assumptions. Known polynomial-time algorithms rely on directional moments, and the assumption that component distributions differ from a Gaussian in some small moment. A related relevant problem, Non-Gaussian Component Analysis (NGCA), aims to extract a hidden non-Gaussian direction in a high-dimensional distribution. Here too, the main idea is the fact that non-Gaussian component must have some finite moment different from that of a Gaussian. While finite moment difference implies a TV distance lower bound, to get  $\epsilon$ -TV distance, one might need to use  $k$ 'th moments for  $k = \Omega (\log (1 / \epsilon))$  even for logconcave densities. As the dependence on the moment number is exponential (even for the sample complexity), this approach does not yield polytime algorithms in terms of TV distance, the natural notion for classification.
The idea of applying Principal component analysis (PCA) to re-weighted samples was used in [BV08] to unravel a mixture of well-separated Gaussians. For a mixture of two general Gaussians that are mean separated, after making the mixture isotropic, it was shown that either the mean or top eigenvector of the covariance of a re-weighted sample reveals the vector of the mean differences. This high-level approach was used for solving general ICA by estimating re-weighted higher moments (tensors) [GVX14]. Higher moment re-weightings were also used by [VX11] to give an algorithm for factoring a distribution and learning "subspace juntas", functions of an unknown low-dimensional subspace, and by [TV18] to give a more efficient algorithm for the special case of NGCA. The question of whether expensive higher moment algorithms could be replaced by re-weighted second moments is natural and one variant was specifically suggested by [TV18] for NGCA. Our work validates this intuition with rigorous polynomial-time algorithms.
Our algorithm first makes the data to be isotropic using the sample mean and sample covariance. Then we apply the weight  $w(y, \alpha) = e^{\alpha \| y \|^2}$  to each isotropized sample  $y$ , and compute the re-weighted mean and the top eigenvector of the re-weighted covariance matrix. Then for each candidate normal vector, we project the data to it, and scan to find the maximum gap. The algorithm outputs the vector with the maximal gap among all candidate vectors. We give the formal description in Algorithm 1.
We demonstrate that Algorithm 1 operates within polynomial time and sample complexity. The details regarding sample complexity are presented in Theorem 1. The time complexity is justified by
Algorithm 1 Unsupervised Halfspace Learning with Contrastive Moments
Input: Unlabeled data  $S = \{x^{(1)},\dots ,x^{(N)}\} \subset \mathbb{R}^d.\epsilon ,\delta >0$
- (Isotropize) Compute the sample mean and covariance:
$$
\hat {\mu} = \frac {1}{N} \sum_ {j = 1} ^ {N} x ^ {(j)}, \qquad \hat {\Sigma} = \frac {1}{N} \sum_ {j = 1} ^ {N} (x ^ {(j)} - \hat {\mu}) (x ^ {(j)} - \hat {\mu}) ^ {\top}.
$$
Make the data isotropic:  $y^{(j)} = \hat{\Sigma}^{-1 / 2}(x^{(j)} - \hat{\mu})$ .
- (Re-weighted Moments) Set  $\alpha_{1} = -c_{1}\epsilon^{82} / d, \alpha_{2} = -c_{2}\epsilon^{42} / d$  and  $\alpha_{3} = -c_{3}\epsilon^{2}$ . Let  $w(y, \alpha) = e^{\alpha \|y\|^2}$ . Compute the re-weighted sample means  $\mu_{\alpha_1}, \mu_{\alpha_2}$  using  $\alpha_{1}, \alpha_{2}$  and the re-weighted sample covariance using  $\alpha_{3}$  as follows:
$$
\tilde {\mu} _ {\alpha_ {i}} = \frac {1}{N} \sum_ {j = 1} ^ {N} w (y ^ {(j)}, \alpha_ {i}) y ^ {(j)}, \text {f o r} i \in \{1, 2 \} \text {a n d} \tilde {\Sigma} = \frac {1}{N} \sum_ {j = 1} ^ {N} w (y ^ {(j)}, \alpha_ {3}) y ^ {(j)} y ^ {(j) ^ {\top}}
$$
Compute the top eigenvector  $v$  of  $\hat{\Sigma}$
- (Max Margin) Calculate the max margin (i.e., maximum gap) of the one-dimensional projections of the data along the vectors  $\hat{\mu}_{\alpha_1}, \hat{\mu}_{\alpha_2}, v$ , and let  $\hat{u}$  be the vector among these with the largest margin.
return  $\hat{u}$
the algorithm's process: it calculates the re-weighted sample mean and the top eigenvector of the re-weighted sample covariance matrix, both of which require polynomial time.
In our algorithm, we consider two cases depending on whether the removed band  $[a,b]$  is origin-symmetric. If it is asymmetric, we will show that one of the re-weighted means with two  $\alpha$ s gives us the correct direction by showing that the re-weighted mean along  $u$  has a gap from zero while the re-weighted mean along all other orthogonal directions is zero. We state the positive gap quantitatively in Lemma 4. Otherwise, if the band is symmetric, we will show a positive gap between the top two eigenvalues of the re-weighted covariance matrix, and the top eigenvector corresponds to  $u$ . We quantify the gap between the top two eigenvalues in Lemma 5. In the algorithm, since we know neither the underlying distribution mean nor the location of the removed band, we have to compute both re-weighted means and re-weighted covariance, and then get the correct direction among all three candidate vectors by calculating the margin and finding the one with the largest margin.
Lemma 4 (Quantitative Gap of Contrastive Mean). Suppose that  $|a + b| \geq \epsilon^5$ . Then, for  $\alpha_1 = -c_1\epsilon^{82} / d, \alpha_2 = -c_2\epsilon^{42} / d$ , the re-weighted mean of  $P$ , denoted as  $\mu_{\alpha_1}$  and  $\mu_{\alpha_2}$ , satisfies
$$
\max  \left(\left| u ^ {\top} \mu_ {\alpha_ {1}} \right|, \left| u ^ {\top} \mu_ {\alpha_ {2}} \right|\right) > \frac {C \epsilon^ {1 5 9}}{d ^ {2}} f o r s o m e c o n s t a n t C > 0,
$$
$$
\forall v \bot u, \quad v ^ {\top} \mu_ {\alpha_ {1}} = v ^ {\top} \mu_ {\alpha_ {2}} = 0.
$$
Lemma 5 (Quantitative Spectral Gap of Contrastive Covariance). Suppose that  $|a + b| < \epsilon^5$ . Choose  $\alpha_3 = -c_3\epsilon^2$  for some constant  $c_3 > 0$ . Then, for an absolute constant  $C$ , the top two eigenvalues  $\lambda_1 \geq \lambda_2$  of the corresponding re-weighted covariance of  $P$  satisfy
$$
\lambda_ {1} - \lambda_ {2} \geq C \epsilon^ {3} \lambda_ {1}.
$$
4 Proofs
Recall that we are given data  $x^{(1)}, \dots, x^{(N)}$  drawn from the affine product distribution with  $\epsilon$ -margin  $\widehat{P}$ . Algorithm 1 first makes the data isotropic. Denote  $y^{(1)}, \dots, y^{(N)}$  as the corresponding isotropicized data. Then each  $y^{(j)}$  is an independent and identically distributed variable drawn from  $P = \hat{q} \otimes q \otimes \dots q$ . Since we compute the re-weighted moments on  $y^{(j)}$  in the algorithm, we analyze the moments of  $P$  directly.
Recall in Definition 1 that  $q$  is the symmetric one-dimensional isotropic logconcave density function, and  $\tilde{q}$  is the density obtained by restricting  $q$  to  $\mathbb{R} \backslash [a, b]$  for some unknown  $a < b$ . Denote  $\mu_1, \sigma_1^2$  as the mean and variance of  $\tilde{q}$ .  $\hat{q}$  is the density obtained after making  $\tilde{q}$  isotropic, with support  $\mathbb{R} \backslash [a', b']$ , where  $a' = \frac{a - \mu_1}{\sigma_1}$ ,  $b' = \frac{b - \mu_1}{\sigma_1}$ . We denote the standard basis of  $\mathbb{R}^d$  by  $\{e_1, \dots, e_d\}$ , and assume wlog that  $e_1 = u$  is the (unknown) normal vector to the band. We write  $x_i := \langle x, e_i \rangle$  as  $x$ 's  $i$ -th coordinate. We assume in our proof that  $|b| > |a|$ . If this condition is not met, we can redefine our interval by setting  $a' = -b$  and  $b' = -a$ . The proof can then be applied considering the distribution is restricted to  $\{x \in \mathbb{R}^d : u^\top x \leq a' \text{ or } u^\top x \geq b'\}$ . For a vector  $x \in \mathbb{R}^d$ , we use  $\| x \|$  to denote its  $l_2$  norm. For a matrix  $A \in \mathbb{R}^{m \times n}$ , we denote its operator norm as  $\| A \|_{\mathrm{op}}$ .
Contrastive Mean. We can write the contrastive mean as a linear combination of exponential functions of  $\alpha$ . By Descartes' rule of signs, the number of zeros of this function is at most two. Since  $\alpha = 0$  is one root and corresponds to mean zero, there is at most one nonzero root. And thus we have that for any two distinct nonzero  $\alpha$ 's, at least one of them achieves nonzero contrastive mean.
Figure 4.1: Coefficient of  $e^{\alpha x^2}$  in  $F(\alpha)$ . In the proof of Lemma 1, by combining  $e^{\alpha x^2}$  terms, we flip  $\hat{q}(x)$  horizontally. For  $a' > 0$ , the coefficient is negative when  $x \in (a', b')$  and non-negative outside the interval. For  $a' \leq 0$ , it is negative when  $x \in (-a', b')$  and positive when  $x > b'$ .
Theorem 3 (Descartes' Rule of Signs in the Integral Form). Let  $F(\alpha) = \int_0^\infty a(x)e^{\alpha x^2}dx$ . Then the number of roots of  $F(\alpha) = 0$  is at most the number of sign changes in  $a(x)$  for  $x \geq 0$ .
Proof of Lemma 1 (Contrastive Mean).  $|b| > |a|$  implies that  $\mu_1 < 0$ . For any  $x \geq 0$ , we have
$$
\hat {q} (x) = \frac {\sigma_ {1} q (x \sigma_ {1} + \mu_ {1})}{1 - \int_ {a} ^ {b} q (x) d x} \geq \frac {\sigma_ {1} q (- x \sigma_ {1} + \mu_ {1})}{1 - \int_ {a} ^ {b} q (x) d x} = \hat {q} (- x).
$$
Since  $P$  is a product distribution, we have
$$
\underset {x \sim P} {\mathbb {E}} e ^ {\alpha \| x \| ^ {2}} x _ {1} = \underset {x _ {1} \sim \hat {q}} {\mathbb {E}} e ^ {\alpha x _ {1} ^ {2}} x _ {1} \cdot \prod_ {i = 2} ^ {d} \underset {x _ {i} \sim q} {\mathbb {E}} e ^ {\alpha x _ {i} ^ {2}}
$$
We denote
$$
F (\alpha) = \underset {x \sim \hat {q}} {\mathbb {E}} e ^ {\alpha x _ {1} ^ {2}} x _ {1} = \int_ {\mathbb {R} \backslash \left[ a ^ {\prime}, b ^ {\prime} \right]} e ^ {\alpha x ^ {2}} x \hat {q} (x) d x \tag {4.1}
$$