Title: Contrastive Moments: Unsupervised Halfspace Learning in Polynomial Time
Venue: NeurIPS 2023
Accepted: yes
Keywords: Unsupervised Learning

Learning Halfspaces

Non-Gaussian Component analysis
Abstract: We give a polynomial-time algorithm for learning high-dimensional halfspaces with margins in $d$-dimensional space to within desired Total Variation (TV) distance when the ambient distribution is an unknown affine transformation of the $d$-fold product of an (unknown) symmetric one-dimensional logconcave distribution, and the halfspace is introduced by deleting at least an $\epsilon$ fraction of the data in one of the component distributions. Notably, our algorithm does not need labels and establishes the unique (and efficient) identifiability of the hidden halfspace under this distributional assumption.  The sample and time complexity of the algorithm are polynomial in the dimension and $1/\epsilon$. The algorithm uses only the first two moments of *suitable re-weightings* of the empirical distribution, which we call *contrastive moments*; its analysis uses classical facts about generalized Dirichlet polynomials and relies crucially on a new monotonicity property of the moment ratio of truncations of logconcave distributions. Such algorithms, based only on first and second moments were suggested in earlier work, but hitherto eluded rigorous guarantees.

Prior work addressed the special case when the underlying distribution is Gaussian via Non-Gaussian Component Analysis. We improve on this by providing polytime guarantees based on TV distance, in place of existing moment-bound guarantees that can be super-polynomial. Our work is also the first to go beyond Gaussians in this setting.

Contrastive Moments: Unsupervised Halfspace Learning in Polynomial Time

Abstract

We give a polynomial-time algorithm for learning high-dimensional halfspaces with margins in d -dimensional space to within desired Total Variation (TV) distance when the ambient distribution is an unknown affine transformation of the d -fold product of an (unknown) symmetric one-dimensional logconcave distribution, and the halfspace is introduced by deleting at least an ϵ fraction of the data in one of the component distributions. Notably, our algorithm does not need labels and establishes the unique (and efficient) identifiability of the hidden halfspace under this distributional assumption. The sample and time complexity of the algorithm are polynomial in the dimension and 1 /ϵ . The algorithm uses only the first two moments of suitable re-weightings of the empirical distribution, which we call contrastive moments ; its analysis uses classical facts about generalized Dirichlet polynomials and relies crucially on a new monotonicity property of the moment ratio of truncations of logconcave distributions. Such algorithms, based only on first and second moments were suggested in earlier work, but hitherto eluded rigorous guarantees.

Prior work addressed the special case when the underlying distribution is Gaussian via Non-Gaussian Component Analysis. We improve on this by providing polytime guarantees based on TV distance, in place of existing moment-bound guarantees that can be super-polynomial. Our work is also the first to go beyond Gaussians in this setting.

1 Introduction

Suppose points in R d are labeled according to a linear threshold function (a halfspace). Learning a threshold function from labeled examples is the archetypal well-solved problem in learning theory, in both the PAC and mistake-bound models; its study has led to efficient algorithms, a range of powerful techniques and many interesting learning paradigms. While the sample complexity in general grows with the dimension, when the halfspace has a margin , the complexity can instead be bounded in terms of the reciprocal of the squared margin width [PCST99, SBS + 00, AV06, LS11]. The problem is also very interesting for special classes of distributions, e.g., when the underlying distribution is logconcave, agnostic learning is possible [KKMS08], and active learning needs fewer samples compared to the general case [BBL06].

The main motivation for our work is learning a halfspace with a margin with no labels , i.e., unsupervised learning of halfspaces. This is, of course, impossible in general - there could be multiple halfspaces with margins consistent with the data - raising the question: Can there be natural distributional assumptions that allow the unsupervised learning of halfspaces? For example, suppose data is drawn from a Gaussian in R d with points in an unknown band removed, i.e., we assume there exists a unit vector u ∈ R d and an interval [ a, b ] so that the input distribution is the Gaussian restricted to the set { x ∈ R d |⟨ u, x ⟩ ≤ a or ⟨ u, x ⟩ ≥ b } . Can the vector u be efficiently learned? Such a distributional assumption ensures that the band normal to u is essentially unique, leaving open the question of whether it can be efficiently learned.

Such models have been considered in the literature, notably for Non-Gaussian Component Analysis (NGCA) [BKS + 06, TV18], learning relevant subspaces [Blu94, VX11] and low-dimensional convex concepts [Vem10] where data comes from a product distribution with all components being Gaussian except for one (or a small number). It is assumed that the non-Gaussian component differs from Gaussian in some low moment and the goal is to identify this component. Another related model is Independent Component Analysis (ICA) where the input consists of samples from an affine transformation of a product distribution and the goal is to identify the transformation itself [Com94, Car98, GVX14, JKV23]. For this problem to be well-defined, it is important that at most one component of the product distribution is Gaussian. No such assumption is needed for NGCA or the more general problem we consider here.

Formally, we consider the following model and problem, illustrated in Figure 1.1.

Definition 1 (Affine Product Distribution with ϵ -Margin) . Let q be a symmetric one-dimensional isotropic logconcave density function. Let Q be the d -fold product distribution obtained from q . Let ˆ q be the isotropized density obtained after restricting q to R \ [ a, b ] where q (( -∞ , a ]) ≥ ϵ, q ([ a, b ]) ≥ ϵ and q ([ b, ∞ )) ≥ ϵ . Let P be the product of one copy of ˆ q and d -1 copies of q . Let ̂ P be obtained by a full-rank affine transformation of P ; we refer to ̂ P as an Affine Product Distribution with ϵ -Margin . Let u be the unit vector normal to the margin before transformation.

!(#)

% &

'

%

&

'

!(#)

(

(

) (

) (

! "

"

! "

#

Transformation

!

!

" = $

!

!

" = %

!

(a) Symmetric and asymmetric margin [ a, b ] . We use contrastive covariance and mean respectively.

(b) Definition 1 with uniform q . ̂ P is a full-rank affine transformation of the product distribution P .

(c) Unlabeled data is drawn from ̂ P . The goal is to learn the normal vector u .

With this model in hand, we have the following algorithmic problem.

Problem. Given input parameters ϵ, δ > 0 and access to iid samples from ̂ P , an affine product distribution with ϵ -margin, the learning problem is to compute a unit vector ˜ u that approximates u to within TV distance δ . That is, the TV distance between the corresponding ˜ P and P is at most δ , where ˜ P is the distribution with margin normal to ˜ u .

In this formulation of the problem with a TV distance guarantee, if each side of the halfspace receives a different label, then the probability that the output halfspace of the data disagrees with the true label (up to swapping the labels) is at most δ .

A natural approach to identifying the halfspace is maximum margin clustering [XNLS04]: find a partition of the data into two subsets s.t. the distance between the two subsets along some direction is maximized. Unfortunately, this optimization problem is NP-hard, even to approximate.

There are at least two major difficulties we have to address. The first is the unknown affine transformation, which we cannot hope to completely identify in general. The second is that, even if we reversed the transformation, the halfspace normal is in an arbitrary direction in R d and would be undetectable in almost all low-dimensional projections, i.e., we have a needle in a haystack problem.

1.1 Results and Techniques

We give an efficient algorithm for the unsupervised halfspace learning problem under any symmetric product logconcave distribution. It consists of the following three high-level steps.

Make the data isotropic (mean zero and covariance matrix identity).

Re-weight data and compute the re-weighted mean ˜ µ i and the top eigenvector v of the re-weighted covariance.

Project data along the vectors ˜ µ i , v , and output the vector with the largest margin.

Although the algorithm is simple and intuitive, its analysis has to overcome substantial challenges. Our main result is the following. The formal version (Theorem 7) is in the Appendix.

Theorem 1 (Main) . There is an algorithm that can learn any affine product distribution with ϵ -margin to within TV distance δ with time and sample complexity that are polynomial in d, 1 /ϵ and 1 /δ with high probability.

To see the idea of the algorithm, we first consider the case when no affine transformation is applied. In this case, we can detect the direction u by calculating the empirical mean and top eigenvector of the empirical uncentered covariance matrix. If the margin [ a, b ] lies on one side of the origin, the mean along u is nonzero while the mean in any other direction that is orthogonal to u is zero. Thus the mean itself reveals the vector u . Otherwise, we can show that the second moment along u is higher than along any other orthogonal direction. Thus, there is a positive gap between the top two eigenvalues of the uncentered covariance matrix and the top eigenvector is u . In fact, the algorithm applies more generally, to the product distribution created from one-dimensional bounded isoperimetric distributions. A one-dimensional distribution p is isoperimetric if there exists a constant ψ > 0 such that for any x ∈ R , p ( x ) ≥ ψ min { p ([ x, ∞ )) , p (( -∞ , x ]) } . The theorem is stated as follows. The formal version (Theorem 6), algorithm and proofs are included in the Appendix.

Theorem 2 (Isotropic Isoperimetric Distribution) . There is an algorithm that can learn any isotropic isoperimetric bounded product distribution with ϵ -margin to within TV distance δ with time and sample complexity that are polynomial in d, 1 /ϵ, 1 /δ with high probability.

In the general case, when an unknown affine transformation is applied, the algorithm first computes the empirical mean and covariance of the sample and makes the empirical distribution isotropic. Then we will consider two cases as illustrated in Figure 1.1a. If the unknown band is not centered around the mean along u , we can expect the empirical mean to differ from the mean of the underlying product distribution without the margin. Consequently, if we knew the latter, we can use the difference to estimate u . However, in general, we do not have this information. Instead, we demonstrate that there exists a re-weighting of the sample so that re-weighted empirical mean compared to the unweighted empirical mean is a good estimate of u . In other words, with appropriate re-weighting, the mean shifts along the normal direction to the unknown band. On the other hand, if the band is centered along u , the mean shift will be zero. In this scenario, we will show that the maximum eigenvector of a re-weighted covariance matrix is nearly parallel to u !

Our algorithm only uses first and second order moments, can be implemented efficiently, and is in fact practical (see Section 5). The main challenges are (1) proving the existence of band-revealing re-weightings and (b) showing that a polynomial-sized sample (and polynomial time) suffice.

To prove the main theorem, we will show that either the re-weighted mean induces a contrastive gap (Lemma 1), or the eigenvalues of the re-weighted covariance matrix induce a contrastive gap (Lemma 2). In the subsequent two lemmas, we adopt the notation from Definition 1. Here, P represents a product distribution with ϵ -margin defined by the interval [ a, b ] (before transformation).

Lemma 1 (Contrastive Mean) . If | a + b | > 0 , then for any two distinct nonzero α 1 , α 2 ∈ R , at least one of the corresponding re-weighted means is nonzero, i.e.,

Lemma 2 (Contrastive Covariance) . If a + b = 0 , then there exists an α < 0 , such that (1) there is a positive gap between the top two eigenvalues of the re-weighted uncentered covariance matrix ˜ Σ = E x ∼ P e α ∥ x ∥ 2 ( xx ⊤ ) . That is, λ 1 ( ˜ Σ) > λ 2 ( ˜ Σ) . (2) The top eigenvector of ˜ Σ is u .

The proof of Lemma 1 uses Descartes' rule of signs applied to a suitable potential function. To prove Lemma 2, we develop a new monotonicity property of the moment ratio (defined as the ratio of the variance of X 2 and the squared mean of X 2 ) for truncations of logconcave distributions. The moment ratio is essentially the square of the coefficient of variation of X 2 . An insight from the monotonicity of the moment ratio is that for logconcave distributions with positive support, when the distribution is restricted to an interval away from the origin, it needs a smaller sample size to estimate its second moment accurately. We state the lemma as follows.

Lemma 3 (Monotonicity of Moment Ratio) . Let q be a logconcave distribution in one dimension with nonnegative support. For any t ≥ 0 , let q t be the distribution obtained by restricting q to [ t, ∞ ) . q , defined as mr q ( t ) := var q t ( X 2 ) E 2 2 , is strictly decreasing with t .

Then the moment ratio of ( q t X )

To obtain polynomial guarantees, we will need quantitative estimates of the inequalities in the above two lemmas. Establishing such quantitative bounds is the bulk of the technical contribution of this paper. While our focus is on proving polynomial bounds, whose existence a priori is far from clear, we did not optimize the polynomial bounds themselves; our experimental results suggest that in fact the dependence on both d and 1 /ϵ might be linear!

1.2 Related Work

Efficient algorithms for supervised halfspace learning [Ros58, MP69], combined with the kernel trick [CST + 00, HDO + 98], serve as the foundation of much of learning theory. Halfspaces with margin are also well-studied, due to their motivation from the brain, attribute-efficient learning [Val98, Blu90], random projection based learning [AV06], and turn out to have sample complexity that grows inverse polynomially with the margin, independent of the ambient dimension. When examples are drawn from a unit Euclidean ball in R d , and the halfspace has margin γ , then the sample complexity grows as O (1 /γ 2 ) regardless of the dimension. This leads to the question of whether labels are even necessary, or the halfspace can be identified from unlabeled samples efficiently - the focus of the present paper.

The model of unsupervised learning we study is similar to other classical models in the literature, notably Independent Component Analysis where input data consists of iid samples from an unknown affine transformation of a product distribution. There, the goal is to recover the affine transformation under minimal assumptions. Known polynomial-time algorithms rely on directional moments, and the assumption that component distributions differ from a Gaussian in some small moment. A related relevant problem, Non-Gaussian Component Analysis (NGCA), aims to extract a hidden non-Gaussian direction in a high-dimensional distribution. Here too, the main idea is the fact that non-Gaussian component must have some finite moment different from that of a Gaussian. While finite moment difference implies a TV distance lower bound, to get ϵ -TV distance, one might need to use k 'th moments for k = Ω(log(1 /ϵ )) even for logconcave densities. As the dependence on the moment number is exponential (even for the sample complexity), this approach does not yield polytime algorithms in terms of TV distance, the natural notion for classification.

The idea of applying Principal component analysis (PCA) to re-weighted samples was used in [BV08] to unravel a mixture of well-separated Gaussians. For a mixture of two general Gaussians that are mean separated, after making the mixture isotropic, it was shown that either the mean or top eigenvector of the covariance of a re-weighted sample reveals the vector of the mean differences. This high-level approach was used for solving general ICA by estimating re-weighted higher moments (tensors) [GVX14]. Higher moment re-weightings were also used by [VX11] to give an algorithm for factoring a distribution and learning 'subspace juntas", functions of an unknown low-dimensional subspace, and by [TV18] to give a more efficient algorithm for the special case of NGCA. The question of whether expensive higher moment algorithms could be replaced by re-weighted second moments is natural and one variant was specifically suggested by [TV18] for NGCA. Our work validates this intuition with rigorous polynomial-time algorithms.

2 Algorithm

Our algorithm first makes the data to be isotropic using the sample mean and sample covariance. Then we apply the weight w ( y, α ) = e α ∥ y ∥ 2 to each isotropized sample y , and compute the re-weighted mean and the top eigenvector of the re-weighted covariance matrix. Then for each candidate normal vector, we project the data to it, and scan to find the maximum gap. The algorithm outputs the vector with the maximal gap among all candidate vectors. We give the formal description in Algorithm 1.

3 Analysis

We demonstrate that Algorithm 1 operates within polynomial time and sample complexity. The details regarding sample complexity are presented in Theorem 1. The time complexity is justified by

Algorithm 1 Unsupervised Halfspace Learning with Contrastive Moments

Input: Unlabeled data S = { x (1) , · · · , x ( N ) } ⊂ R d . ϵ, δ > 0 .

(Isotropize) Compute the sample mean and covariance:

Make the data isotropic: y ( j ) = ˆ Σ - 1 / 2 ( x ( j ) - ˆ µ )

(Re-weighted Moments) Set α 1 = -c 1 ϵ 82 /d, α 2 = -c 2 ϵ 42 /d and α 3 = -c 3 ϵ 2 . Let w ( y, α ) = e α ∥ y ∥ 2 . Compute the re-weighted sample means µ α 1 , µ α 2 using α 1 , α 2 and the re-weighted sample covariance using α 3 as follows:

Compute the top eigenvector v of ˜ Σ .

(Max Margin) Calculate the max margin (i.e., maximum gap) of the one-dimensional projections of the data along the vectors ˆ µ α 1 , ˆ µ α 2 , v , and let ˆ u be the vector among these with the largest margin.

return ˆ u .

the algorithm's process: it calculates the re-weighted sample mean and the top eigenvector of the re-weighted sample covariance matrix, both of which require polynomial time.

In our algorithm, we consider two cases depending on whether the removed band [ a, b ] is originsymmetric. If it is asymmetric, we will show that one of the re-weighted means with two α s gives us the correct direction by showing that the re-weighted mean along u has a gap from zero while the re-weighted mean along all other orthogonal directions is zero. We state the positive gap quantitatively in Lemma 4. Otherwise, if the band is symmetric, we will show a positive gap between the top two eigenvalues of the re-weighted covariance matrix, and the top eigenvector corresponds to u . We quantify the gap between the top two eigenvalues in Lemma 5. In the algorithm, since we know neither the underlying distribution mean nor the location of the removed band, we have to compute both re-weighted means and re-weighted covariance, and then get the correct direction among all three candidate vectors by calculating the margin and finding the one with the largest margin.

Lemma 4 (Quantitative Gap of Contrastive Mean) . Suppose that | a + b | ≥ ϵ 5 . Then, for α 1 = -c 1 ϵ 82 /d, α 2 = -c 2 ϵ 42 /d , the re-weighted mean of P , denoted as µ α 1 and µ α 2 , satisfies

Lemma 5 (Quantitative Spectral Gap of Contrastive Covariance) . Suppose that | a + b | < ϵ 5 . Choose α 3 = -c 3 ϵ 2 for some constant c 3 > 0 . Then, for an absolute constant C , the top two eigenvalues λ 1 ≥ λ 2 of the corresponding re-weighted covariance of P satisfy

4 Proofs

Recall that we are given data x (1) , · · · , x ( N ) drawn from the affine product distribution with ϵ -margin ̂ P . Algorithm 1 first makes the data isotropic. Denote y (1) , · · · , y ( N ) as the corresponding isotropicized data. Then each y ( j ) is an independent and identically distributed variable drawn from P = ˆ q ⊗ q ⊗··· q . Since we compute the re-weighted moments on y ( j ) in the algorithm, we analyze the moments of P directly.

Recall in Definition 1 that q is the symmetric one-dimensional isotropic logconcave density function, and ˜ q is the density obtained by restricting q to R \ [ a, b ] for some unknown a < b . Denote µ 1 , σ 2 1 as the mean and variance of ˜ q . ˆ q is the density obtained after making ˜ q isotropic, with support R \ [ a ′ , b ′ ] , where a ′ = a -µ 1 σ 1 , b ′ = b -µ 1 σ 1 . We denote the standard basis of R d by { e 1 , · · · , e d } , and assume wlog that e 1 = u is the (unknown) normal vector to the band. We write x i := ⟨ x, e i ⟩ as x 's i -th coordinate. We assume in our proof that | b | > | a | . If this condition is not met, we can redefine our interval by setting a ′ = -b and b ′ = -a . The proof can then be applied considering the distribution is restricted to { x ∈ R d : u ⊤ x ≤ a ′ or u ⊤ x ≥ b ′ } . For a vector x ∈ R d , we use ∥ x ∥ to denote its l 2 norm. For a matrix A ∈ R m × n , we denote its operator norm as ∥ A ∥ op.

Contrastive Mean. We can write the contrastive mean as a linear combination of exponential functions of α . By Descartes' rule of signs, the number of zeros of this function is at most two. Since α = 0 is one root and corresponds to mean zero, there is at most one nonzero root. And thus we have that for any two distinct nonzero α 's, at least one of them achieves nonzero contrastive mean.

+

#

> -

+

#

≤ -

)

%

#′

&′

)

%

-#′

&′

)

#′

#′

%

&′

-#′

"

!(-))

"

!())

)

#′

%

&′

"

!(-))

"

!())

Theorem 3 (Descartes' Rule of Signs in the Integral Form) . Let F ( α ) = ∫ ∞ 0 a ( x ) e αx 2 dx . Then the number of roots of F ( α ) = 0 is at most the number of sign changes in a ( x ) for x ≥ 0 .

Proof of Lemma 1 (Contrastive Mean). | b | > | a | implies that µ 1 < 0 . For any x ≥ 0 , we have

Since P is a product distribution, we have

Then we rearrange F ( α ) by combining e αx 2 as in Figure 4.1.

If a ′ ≤ 0 , we rewrite F ( α ) as

We treat F ( α ) as the integral of a ( x ) e αx 2 for x ≥ -a ′ . Since ˆ q ( x ) -ˆ q ( -x ) > 0 for x > b ′ , we have a ( x ) > 0 for x ∈ [ -a ′ , b ′ ) and a ( x ) < 0 for x > b ′ . In other words, for increasing x , the sign of a ( x ) only changes once. By Theorem 3, F ( α ) = 0 has at most one root.

We denote

If a ′ > 0 , we arrange F ( α ) in the same way and get

Similarly, we treat F ( α ) as the integral of a ( x ) e αx 2 for x ≥ 0 . For increasing x , the sign of a ( x ) changes twice. By Descartes' rule of signs, F ( α ) = 0 has at most two roots. In addition, we know F (0) = E P x 1 = 0 by definition of P , which implies that α = 0 is one root of F ( α ) . So there is at most one nonzero root of F ( α ) . In other words, for any two distinct nonzero α 1 , α 2 , at least one of F ( α 1 ) , F ( α 2 ) is nonzero. Consequently,

̸

Proof Idea of Lemma 4 (Quantitative Gap of Contrastive Mean). To get a quantitative bound on the contrastive mean, we follow the same idea of bounding the number of roots of F ( α ) as in Lemma 1. By taking the derivative of F ( α ) , we can show that either F ′ (0) = 0 or F ′′ (0) = 0 . Then by Taylor expansion, we can choose two distinct α 's (near zero) so that one of the corresponding contrastive means is bounded away from zero.

̸

Moment Ratio and Contrastive Covariance. To prove Lemma 2, we develop a new monotonicity property of the moment ratio of logconcave distributions. Moment ratio is specifically defined as the ratio of the fourth moment to the square of the second moment of truncated versions of the distribution. This measurement essentially reflects the uncentered kurtosis of the distribution. We will prove the monotonicity of the moment ratio by reducing the case of general logconcave distributions to exponential distributions.

Proof of Lemma 3 (Moment Ratio). We show the monotonicity of moment ratio by showing its derivative with respect to t is negative. Define M k ( t ) = ∫ ∞ t x k q ( x ) dx . By calculation, the derivative of moment ratio is proportional to -H ( t ) , where H ( t ) is defined as follows:

Let h ( x ) = βe -γx be an exponential function ( β, γ > 0 ) such that

Then we have

By the logconcavity of q , the graph of h intersects with the graph of q at exactly two points u ′ < v , where v > 0 . Also we have h ( x ) ≤ q ( x ) at the interval [ u ′ , v ] and h ( x ) > q ( x ) outside the interval. Let u = max { 0 , u ′ } . So for x ≥ 0 , ( x -u )( x -v ) has the same sign as h ( x ) -q ( x ) . Since t ≥ 0 , we have

Expanding the terms and we get

This shows that N 4 ( t ) ≥ M 4 ( t ) . To show that H ( t ) > 0 , we consider two cases.

Firstly if M 2 ( t ) -2 t 2 ≥ 0 , we have

Secondly if M ( t ) -2 t 2 < 0 , by calculation of the exponential function's moments, we have

This implies that

The equality holds if and only if M 4 ( t ) = N 4 ( t ) > 0 . Then we have

Combining both cases, mr ′ q ( t ) < 0 , ∀ t ≥ 0 , which implies that the moment ratio of q is strictly decreasing with respect to t .

Proof Idea of Lemma 2 (Constrastive Covariance). View the spectral gap of the re-weighted covariance, denoted as λ 1 ( ˜ Σ) -λ 2 ( ˜ Σ) , as a function S ( α ) . By calculation, S (0) = 0 and S ′ (0) is proportional to mr q ( b ) -mr q (0) , which is negative by the monotonicity property of moment ratio. Then we can prove the spectral gap for the chosen α using Taylor expansion.

Proof Idea of Lemma 5 (Quantitative Gap of Contrastive Covariance). We first prove the case when a = -b , and then extend the result to the near-symmetric case when | a + b | < ϵ 5 by comparing the re-weighted second moments of two distributions created by restricting q to R \ [ a, b ] and R \ [ -b, b ] respectively.

The following theorem enables us to bound the sample complexity to estimate the covariance matrix. Lemma 6 (Covariance Estimation [SV13]) . Consider independent isotropic random vectors X i in R d s.t. for some C, η > 0 , for every orthogonal projection P in R d ,

Let ϵ ∈ (0 , 1) . Then with the sample size N = O ( dϵ -2 -2 /η ) , we have

The following classical theorem will allow us to use the eigenvalue gap to identify the relevant vector.

Lemma 7 (Davis-Kahan [DK70]) . Let S and T be symmetric matrices with the same dimensions. For a fixed i , assume that the largest eigenvalue of S is well separated from the second largest eigenvalue of S , i.e. ∃ δ > 0 s.t. λ 1 ( S ) -λ 2 ( S ) > δ . Then for the top eigenvectors of S and T , we have

Proof Sketch of Main Theorem. We prove the theorem by considering whether the removed band [ a, b ] is symmetric or not. If | a + b | ≥ ϵ 5 , by Lemma 4, for the chosen two α s, at least one of the contrastive means ˜ µ i along u direction is bounded away from zero (by poly (1 /d, ϵ ) ) and the projection of the contrastive mean along any direction orthogonal to u is zero by symmetry. So by Chebyshev's Inequality, we will ensure that the angle between u and ˜ µ i is less than δ with high probability using O ( poly ( d, 1 /ϵ, 1 /δ )) samples. On the other hand, if | a + b | < ϵ 5 , we rely on the contrastive covariance. By Lemma 5, the top eigenvector v aligns with u while the top two eigenvalues λ 1 > λ 2 satisfy λ 1 -λ 2 > poly ( ϵ ) . By Lemma 7, we can upper bound the angle between u and v by the quotient of the operator norm of the difference of the contrastive covariance and sample contrastive covariance and the spectral gap λ 1 -λ 2 . The covariance matrix itself can be estimated to desired accuracy efficiently with O ( poly ( d, 1 /ϵ, 1 /δ )) samples by Lemma 6. This ensures the closeness of u and top eigenvector v and hence a TV-distance guarantee. The formal statement of the theorem and the proofs are included in the Appendix.

5 Experiments

While our primary goal is to establish polynomial bounds on the sample and time complexity, our algorithms are natural and easy to implement. We study the efficiency and performance of Algorithm 1 on data drawn from affine product distributions with margin. Here we consider three special cases of logconcave distribution: Gaussian, uniform in an interval and exponential. We include four experiments. In all results, we measure the performance of the algorithm using the sin of the angle between the true normal vector u and the predicted vector ˆ u , i.e., sin θ ( u, ˆ u ) , which bounds the TV distance between the underlying distribution and the predicted one after isotropic transformation. Experimental results strongly suggest that the sample complexity is a small polynomial, perhaps even just nearly linear in both the dimension and the separation parameter ϵ .

Overall Performance. Here we conduct the experiments based on a grid search of ( a, b ) pairs on three special cases of logconcave distribution: Gaussian, uniform in an interval and exponential. We measure the performance of ( a, b ) pairs, where for each pair of ( a, b ) , we conduct five independent trials. For Gaussian and exponential distribution, we choose -3 ≤ a < b ≤ 3 and for uniform distribution, we choose -1 . 5 ≤ a < b ≤ 1 . 5 . Here we set the dimension d = 10 and sample size N = 1000000 . For the parameters, we choose α 1 = α 3 = -0 . 1 , α 2 = -0 . 2 . See Figure 5.1 as the heatmap of sin θ ( u, ˆ u ) given different pairs of ( a, b ) .

Although in Algorithm 1, we use extremely small values of the weight parameter α , our experiments show that larger constant values also work empirically, leading to much smaller sample complexity. This coincides with our qualitative lemmas (Lemma 1, Lemma 2).

The algorithm performs well as seen in the results, except when a and b are both close to the edge, and thus there is almost no mass on one side of the band. Also, the uniform distribution is the easiest to learn, while the exponential is the hardest among these three distributions. As shown in all three plots, the algorithm performs the best when a and b are near symmetric with origin. In other words, contrastive covariance has better sample complexity than contrastive mean when we fix other hyperparameters. This coincides with our sample complexity bounds as in the proof of Theorem 1.

(b) Uniform

(c) Exponential

-2.7

0.10

-1.35

0.10

-2.7

0.10

-2.1

-1.05

-2.1

-1.5

0.08

-0.75

0.08

-1.5

0.08

-0.9

-0.45

-0.9

-0.3

0.06

-0.15

0.06

-0.3

0.06

b

0.3

b

0.15

b

0.3

0.9

0.04

0.45

0.04

0.9

0.04

1.5

0.75

1.5

2.1

0.02

1.05

0.02

2.1

0.02

2.7

1.35

2.7

m

0

4

2

1

8

2

1

6

0

a

0

0

6

O

2

8

4

0.00

5

1

2

1

の

6

m

a

m

6

0

6

0

0.00

m

4

2

8

1

2

1

0

6

a

0

6

2

C

4

2

0.00

Performance of Contrastive Mean and Covariance. In this experiment, we fix a negative a as the left endpoint of the removed band, and measure the performance of both contrastive mean and contrastive covariance with respect to different margin right endpoint b . As shown in Figure 5.2, contrastive mean performs well except when a + b is close to zero, while contrastive covariance performs well only when a + b is close to zero. This coincides with our algorithm and analysis for the two cases. In addition, our algorithm chooses the best normal vector among candidates from both contrastive mean and covariance. So our algorithm achieves good performance (minimum of contrastive mean and covariance curves).

Specifically, we choose a = -2 , b ∈ [ -1 . 9 , 4] for Gaussian and exponential cases, and a = -0 . 5 , b ∈ [ -0 . 4 , 0 . 9] for uniform case. We choose the dimension d = 10 , the sample size N = 2000000 . We choose α 1 = α 3 = -0 . 1 , α 2 = -0 . 2 . We average the result with 50 independent trials.

2

1

0

1

2

3

4

b

0.0

0.2

0.4

0.6

0.8

1.0

(

u

,

u

)

Contrastive Mean

Contrastive Covariance

a

=

-

2

.

0.4

0.2

0.0

0.2

0.4

0.6

0.8

b

0.0

0.2

0.4

0.6

0.8

1.0

(

u

,

u

)

Contrastive Mean

Contrastive Covariance

(b) Uniform,

a

=

-

0

.

5

.

2

1

0

1

2

3

4

b

0.0

0.2

0.4

0.6

0.8

1.0

(

u

,

u

)

Contrastive Mean

Contrastive Covariance

(c) Exponential,

a

=

-

2

.

Dimension Dependence. In this experiment, we show the relationship between the input dimension d and the sample complexity. For fixed number size N = 1000000 , we measure the performance of our algorithm with different d . The result is averaged based on a grid search of ( a, b ) pairs, where for each pair of ( a, b ) , we conduct five independent trials. For Gaussian and exponential distributions, we choose -3 ≤ a < b ≤ 3 and for uniform distribution, we choose -0 . 8 ≤ a < b ≤ 0 . 8 .

As shown Figure 5.3, the performance scales linearly with growing dimension d , suggesting a linear relationship between the sample complexity and the input dimension.

20

40

60

80

100

d

0.0

0.1

0.2

0.3

0.4

0.5

0.6

(

u

,

u

)

20

40

60

80

100

d

0.0

0.1

0.2

0.3

0.4

(

u

,

u

)

(b) Uniform

20

40

60

80

100

d

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

(

u

,

u

)

(c) Exponential

ϵ -Dependence. To further understand the dependence on the separation parameter ϵ , we plot the performance versus 1 /ϵ in Figure 5.4. Here we calculate 1 /ϵ as 1 /q ([ a, b ]) , and the performance as the median sin θ ( u, ˆ u ) for specific mass q ([ a, b ]) . As we can see the performance drops near linearly with respect to 1 /ϵ , which indicates that the sample complexity is possibly linear in 1 /ϵ as well.

(b) Uniform

(c) Exponential

0.06

0.12

0.10

0.05

0.10

0.08

0.08

0.04

0.03

0.06

sin6

'n）

0.06

0.04

0.04

0.02

0.02

0.02

0.01

2

3

4

5

1/mass

6

7

8

6

10

2

4

1/mass

6

8

10

12

0

2

4

1/mass

6

8

10

6 Discussion

We have proposed and analyzed an efficient algorithm to learn the symmetric product logconcave distribution without labeled data. This is also connected to self-supervised learning. Specifically, contrastive learning without data augmentation is closely related to the contrastive covariance part in our algorithm.

The algorithm delivers more than the theoretical analysis. While our analysis focuses on specific values of α , as demonstrated by the qualitative lemmas (Lemma 1, Lemma 2), any distinct pair of nonzero α values should work for contrastive mean, and any bounded small α should work for contrastive covariance. This flexibility ensures the applicability of the algorithm in various real-world scenarios. Furthermore, our experimental results align with this claim.

The experiments reveals a linear relationship with the input dimension d , raising an open question regarding the improvement of the sample complexity bound to be linear with respect to d (as well as 1 /ϵ ). Additionally, it might be possible to extend the algorithm's application to more general distributions.

Acknowledgements. This work was supported in part by NSF awards CCF-2007443 and CCF2134105 and an ARC fellowship.

References

[AV06] Rosa I Arriaga and Santosh Vempala. An algorithmic theory of learning: Robust concepts and random projection. Machine learning , 63:161-182, 2006.

[BBL06] Maria-Florina Balcan, Alina Beygelzimer, and John Langford. Agnostic active learning. In Proceedings of the 23rd international conference on Machine learning , pages 65-72, 2006.

[BKS + 06] Gilles Blanchard, Motoaki Kawanabe, Masashi Sugiyama, Vladimir Spokoiny, KlausRobert Müller, and Sam Roweis. In search of non-gaussian components of a highdimensional distribution. Journal of Machine Learning Research , 7(2), 2006.

[Blu90] Avrim Blum. Learning boolean functions in an infinite attribute space. In Proceedings of the twenty-second annual ACM symposium on Theory of computing , pages 64-72, 1990.

[Blu94] Avrim Blum. Relevant examples and relevant features: Thoughts from computational learning theory. In AAAI Fall Symposium on Relevance , volume 5, page 1, 1994.

[BV08] S Charles Brubaker and Santosh S Vempala. Isotropic pca and affine-invariant clustering. Building Bridges: Between Mathematics and Computer Science , pages 241-281, 2008.

[Car98] J-F Cardoso. Multidimensional independent component analysis. In Proceedings of the 1998 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP'98 (Cat. No. 98CH36181) , volume 4, pages 1941-1944. IEEE, 1998.

[Com94] Pierre Comon. Independent component analysis, a new concept? Signal processing , 36(3):287-314, 1994.

[CST + 00] Nello Cristianini, John Shawe-Taylor, et al. An introduction to support vector machines and other kernel-based learning methods . Cambridge university press, 2000.

[DK70] Chandler Davis and William Morton Kahan. The rotation of eigenvectors by a perturbation. iii. SIAM Journal on Numerical Analysis , 7(1):1-46, 1970.

[GVX14] Navin Goyal, Santosh Vempala, and Ying Xiao. Fourier pca and robust tensor decomposition. In Proceedings of the forty-sixth annual ACM symposium on Theory of computing , pages 584-593, 2014.

[HDO + 98] Marti A. Hearst, Susan T Dumais, Edgar Osuna, John Platt, and Bernhard Scholkopf. Support vector machines. IEEE Intelligent Systems and their applications , 13(4):18-28, 1998.

[JKV23] He Jia, Pravesh K Kothari, and Santosh S Vempala. Beyond moments: Robustly learning affine transformations with asymptotically optimal error. arXiv preprint arXiv:2302.12289 , 2023.

[KKMS08] Adam Tauman Kalai, Adam R Klivans, Yishay Mansour, and Rocco A Servedio. Agnostically learning halfspaces. SIAM Journal on Computing , 37(6):1777-1805, 2008.

[LS11] Phil Long and Rocco Servedio. Algorithms and hardness results for parallel large margin learning. Advances in Neural Information Processing Systems , 24, 2011.

[LV07] László Lovász and Santosh Vempala. The geometry of logconcave functions and sampling algorithms. Random Structures & Algorithms , 30(3):307-358, 2007.

[MP69] Marvin Minsky and Seymour Papert. Perceptrons: An Introduction to Computational Geometry . MIT Press, Cambridge, MA, USA, 1969.

[PCST99] John Platt, Nello Cristianini, and John Shawe-Taylor. Large margin dags for multiclass classification. Advances in neural information processing systems , 12, 1999.

[Ros58] Frank Rosenblatt. The perceptron: a probabilistic model for information storage and organization in the brain. Psychological review , 65(6):386, 1958.

[SBS + 00] Alexander J Smola, Peter J Bartlett, Dale Schuurmans, Bernhard Schölkopf, et al. Advances in large margin classifiers . MIT press, 2000.

[SV13] Nikhil Srivastava and Roman Vershynin. Covariance estimation for distributions with 2 + ϵ moments. The Annals of Probability , 41(5):3081-3111, 2013.

[TV18] Yan Shuo Tan and Roman Vershynin. Polynomial time and sample complexity for nongaussian component analysis: Spectral methods. In Conference On Learning Theory , pages 498-534. PMLR, 2018.

[Val98] Leslie G Valiant. Projection learning. In Proceedings of the eleventh annual conference on Computational learning theory , pages 287-293, 1998.

[Vem10] Santosh S Vempala. Learning convex concepts from gaussian distributions with pca. In 2010 IEEE 51st Annual Symposium on Foundations of Computer Science , pages 124-130. IEEE, 2010.

[VX11] Santosh S Vempala and Ying Xiao. Structure from local optima: Learning subspace juntas via higher order pca. arXiv preprint arXiv:1108.3329 , 2011.

[XNLS04] Linli Xu, James Neufeld, Bryce Larson, and Dale Schuurmans. Maximum margin clustering. Advances in neural information processing systems , 17, 2004.

Appendix

A Preliminaries

A.1 Logconcave Distributions

Lemma8 (Lemma 5.4, [LV07]) . Let X be a random point drawn from a one-dimensional logconcave distribution. Then

Lemma 9 (Lemma 5.5,[LV07]) . Let p : R → R + be an isotropic logconcave density function. Then we have

(a) For all x , g ( x ) ≤ 1 .

Lemma 10 (Lemma 5.6,[LV07]) . Let X be a random point drawn from a logconcave density function p : R → R + . Then for every c ≥ 0 ,

Lemma 11 (Lemma 5.7, [LV07]) . Let X be a random variable drawn from a logconcave distribution in R . Assume that E X 2 ≤ 1 . Then for any t ≥ 1 , we have

Corollary 1. Let X be a random point drawn from an isotropic symmetric logconcave density function p : R → R + . Then we have for t ≥ 0 , we have

Proof. Since p ( x ) is symmetric, we know p ( x ) is monotonically decreasing for x ≥ 0 . Then we apply Lemma 10 with c = p ( x ) , and get

On other hand, by Lemma 9, we have max x p ( x ) ≥ p (0) ≥ 1 / 8 . So we have

Lemma 12 (Theorem 5.22, [LV07]) . For a random point X drawn from a logconcave distribution in R d , then

Lemma 13. Let X be a random point drawn from an isotropic symmetric logconcave density function p : R → R + . Then for any t ≥ 3 , we have

Proof. First we claim that p (3) < p (0) / 2 . Otherwise,

This leads to the contradiction. Then for any t ≥ 3 , from the logconcavity of p ,

This implies that

A.2 Descartes' Rule of Signs

Descartes' rule of signs is a well-known principle in algebra that offers a way to estimate the maximum number of real roots for any polynomial. This classical theorem can be stated as follows:

Theorem 4 (Descartes' Rule of Signs) . For the generalized Dirichlet polynomial

the number of roots of F ( x ) = 0 is at most the number of sign changes in the series { a 1 , a 2 , · · · , a n } .

In this section, we state and prove a variant of Descartes' rule of signs in the integral form, which we apply directly to prove Lemma 1. To begin with, we say a function f has a root of order k at point x if

̸

We denote Z ( f ) as the number of roots of f , counted with their orders. Then we can show that the number of roots of f is upper bounded by one plus the number of roots of f ′ in Lemma 14. We use Rolle's Theorem in the proof of the lemma.

Theorem 5 (Rolle's Theorem) . Suppose that a function f is differentiable at all points of interval [ a, b ] and f ( a ) = f ( b ) . Then there is at least one point x 0 ∈ ( a, b ) such that f ′ ( x 0 ) = 0 .

Lemma 14. Z ( f ) ≤ Z ( f ′ ) + 1 .

Proof. Suppose f has a root of order k r as x r , 1 ≤ r ≤ n . Then f ′ has a root of order k -1 at x r . These add up to

By Rolle's Theorem, f ′ ( x ) also has at least n -1 roots in the gaps between the points x r . Together, these two facts give

Theorem 3 (Descartes' Rule of Signs in the Integral Form) . Let F ( α ) = ∫ ∞ 0 a ( x ) e αx 2 dx . Then the number of roots of F ( α ) = 0 is at most the number of sign changes in a ( x ) for x ≥ 0 .

Proof. We prove the theorem with induction on the number of sign changes of a ( x ) . For the base case when the number of sign changes of a ( x ) is zero, we assume wlog that a ( x ) ≥ 0 . Then F ( α ) > 0 and thus F ( α ) has no root. Now we assume that the theorem holds when the number of sign changes is t and we will show the t +1 case.

Let one of the sign changes of a ( x ) occurs at x 0 . Define

which has the same roots as F ( α ) . By taking derivative, we get

Let b ( x ) = a ( x )( x 2 -x 2 0 ) be the new sequence. Then b ( x ) has one less sign changes than a ( x ) . By induction hypothesis, the number of roots of F ′ 0 ( α ) is upper bounded by the number of sign changes of b ( x ) . By Lemma 14, the number of roots of F 0 is upper bounded by the number of sign changes of a ( x ) , thus leading to the induction step.

B Warm-up: Isotropic Isoperimetric Distribution with ϵ -Margin

As a warm-up, we consider the isotropic product distributions with ϵ -margin. Notably, without applying an unknown transformation on data, we can extend the logconcave distributions to isoperimetric distributions. In this section, we will demonstrate how to retrieve the normal vector u by calculating the empirical mean and top eigenvector of the empirical uncentered covariance matrix. This technique is similar to Principal Component Analysis (PCA), but instead of computing covariance matrix, we use the uncentered covariance matrix.

Definition 2. A distribution p with support R is ψ -isoperimetric if there exists ψ > 0 such that for any x ∈ R , we have p ( x ) ≥ ψ min { p ([ x, ∞ )) , p (( -∞ , x ]) } .

Definition 3 (Isotropic Isoperimetric Distribution with ϵ -Margin) . Let q 1 , . . . , q d be symmetric one-dimensional isotropic ψ -isoperimetric density functions bounded by τ . Let Q = q 1 ⊗··· ⊗ q d . Let ˆ q be the density obtained after restricting q 1 to R \ [ a, b ] where q 1 (( -∞ , a ]) ≥ ϵ, q 1 ([ a, b ]) ≥ ϵ and q 1 ([ b, ∞ )) ≥ ϵ . Let P be an arbitrary rotation of ˆ q ⊗ q 2 ⊗··· ⊗ q d . We refer P as an Isotropic Isoperimetric Distribution with ϵ -Margin . Let u be the unit vector normal to the margin.

Problem. Given input parameters ϵ, δ > 0 and access to iid samples from P , an isotropic isoperimetric distribution with ϵ -margin, the learning problem is to compute a unit vector ˜ u that approximates u to within TV distance δ . That is, the TV distance between the corresponding ˜ P and P is at most δ , where ˜ P is the distribution with margin normal to ˜ u .

B.1 Algorithm

Given data drawn from P , we compute the sample mean and the top eigenvector of the uncentered covariance matrix. Then we compare the max margin along these two candidate normal vectors. This gives an efficient algorithm for the problem with no re-weighting. We state the algorithm formally in Algorithm 2.

Algorithm 2 Unsupervised Halfspace Learning from Isotropic Isoperimetric Data

Compute the sample mean and uncentered covariance matrix:

Compute ˆ Σ 's top eigenvector v .

Calculate the max margin (i.e., maximum gap) of the one-dimensional projections of the data along the vectors ˆ µ, v . Let ˆ u be the vector among these two with a larger margin.

return the vector ˆ u .

B.2 Analysis

We demonstrate that Algorithm 2 operates within polynomial time and sample complexity. The details regarding sample complexity are presented in Theorem 2 (formal statement of Theorem 6). The time complexity is justified by the algorithm's process: it calculates the sample mean and the top eigenvector of the sample covariance matrix, both of which require polynomial time.

Theorem 6 (Sample Complexity for Isotropic Isoperimetric Distribution) . Algorithm 2 with N = ˜ O ( d 2 ϵ -6 δ -2 ξ -1 ) samples learns the target isotropic isoperimetric distribution with ϵ -margin to within TV distance δ with probability 1 -ξ .

The analytical approach is straightforward. Given that the component distributions are isotropic, the empirical mean will reveal the band if the removed band [ a, b ] stays on one side of the origin. Otherwise, when [ a, b ] spans across the origin, the variance along the component with the deleted band will increase. Consequently, this component emerges as the top principal component. Intriguing, this

property is 'opposite" to the method used to identify low-dimensional convex concepts in [Vem10]. The latter relies on the Brascamp-Lieb inequality, where the variance of a restricted Gaussian is less than that of the original Gaussian.

To prove Theorem 6, we aim to quantify either the mean gap or the spectral gap (gap between the top two eigenvalues) of the uncentered covariance matrix. Specifically, Lemma 15 indicates that when 0 ≤ a < b , the mean along the direction u significantly deviates from zero. Meanwhile, Lemma 16 demonstrates that when a ≤ 0 < b , there's a gap between the first and second eigenvalues of the uncentered covariance matrix. Subsequently, we employ Lemma 6 [SV13] to determine the sample complexity, and utilize the Davis-Kahan Theorem [DK70] (Lemma 7) to leverage the eigenvalue gap in identifying the pertinent vector u . We leave the proof of the lemmas in Section B.3.

For any x ∈ R , we denote x i as its i -th coordinate. We use ∥ x ∥ to denote its l 2 norm. For a matrix A ∈ R m × n , we denote its operator norm as ∥ A ∥ op. We denote the standard basis of R d by { e 1 , · · · , e d } , and assume wlog that e 1 = u is the (unknown) normal vector to the band. Denote Σ = E x ∼ P xx ⊤ as the uncentered covariance matrix of P , with eigenvalues λ 1 ≥ λ 2 ≥ · · · ≥ λ d .

Lemma 15 (Mean Gap) . For 0 ≤ a < b and b -a ≥ cϵ for constant c > 0 , we have

Lemma 16 (Spectral Gap of Covariance) . If a ≤ 0 < b and b > cϵ for constant c > 0 , then the first and second eigenvalues of the uncentered covariance matrix Σ have the following gap

Furthermore, the top eigenvector corresponds to u .

Now we are ready to prove Theorem 6.

Proof of Theorem 6. We can proceed with the assumption that | b | > | a | . If this condition is not met, we can redefine our interval by setting a ′ = -b and b ′ = -a . The proof can then be applied considering the distribution is restricted to { x ∈ R d : u ⊤ x ≤ a ′ or u ⊤ x ≥ b ′ } . We will prove the theorem by considering two cases: 0 ≤ a < b and a ≤ 0 < b .

We first consider the case when 0 ≤ a < b . Given that q 1 is bounded by τ , it follows that b -a > ϵ/τ . By Lemma 15, we know

Given data x (1) , · · · , x ( N ) , let ˆ µ = 1 N ∑ N j =1 x ( j ) be the sample mean. Then by Chebyshev's Inequality,

Let 0 < ξ < 1 . So we know with sample size N 1 = 16 τ 4 d 2 ϵ 7 δ 2 ψ 2 ξ ,

Then we have while for i ≥ 2 , we have

Secondly, we consider the case where a ≤ 0 < b . Given that b -a > ϵ/τ and | b | > | a | , it results in b > ϵ/ (2 τ ) . By Lemma 16, the top two eigenvalues of Σ , denoted as λ 1 and λ 2 satisfies

By Lemma 6, with sample size N 2 = ˜ O ( dϵ -2 1 ) , with probability at least 1 -ξ ,

By Lemma 7, we know for the top eigenvector v of ˆ Σ satisfies

Choose ϵ 1 = Cϵ 3 δ/ 2 , and we will get sin θ ( e 1 , v ) ≤ δ . The sample size we need is N 2 = ˜ O ( dϵ -6 δ -2 ) . So with sample size N = max( N 1 , N 2 ) = ˜ O ( d 2 ϵ -6 δ -2 ξ -1 ) , Algorithm 2 can recover e 1 within TV distance δ with probability 1 -ξ .

B.3 Proofs

Lemma 15 (Mean Gap) . For 0 ≤ a < b and b -a ≥ cϵ for constant c > 0 , we have

Proof. Since q 1 is ψ -isoperimetric, ∀ t ∈ [ a, b ] , q 1 ( t ) ≥ ψ ∫ ∞ t q 1 ( x ) dx ≥ ψϵ . Then we have

By the definition of expectation, we have

On the other hand, we can calculate the variance as follows.

Lemma 17 (Second Moment) . For a, b satisfying a ≤ 0 < b, b > cϵ for constant c > 0 , we have

E

x

∼

ˆ

q

Proof. By definition of ˆ q , we know its density on the support x ∈ R \ [ a, b ] is

Then we calculate its second moment as follows.

Define g ( x ) := ∫ ∞ x ( t 2 -1) q 1 ( t ) dt, x ≥ 0 . Its derivative is g ′ ( x ) = (1 -x 2 ) q 1 ( x ) . So we know g ( x ) is monotonically increasing when x ∈ [0 , 1] , and decreasing when x ≥ 1 . Since P 1 is symmetric and isotropic, we know ∫ ∞ 0 q 1 ( x ) dx = ∫ ∞ 0 x 2 q 1 ( x ) dx = 1 / 2 . So we have g (0) = 0 . This derives that g ( x ) ≥ 0 , ∀ x ≥ 0 . In other words, ∫ ∞ -a x 2 q 1 ( x ) dx ≥ ∫ ∞ -a q 1 ( x ) dx .

x

2

>

1 +

2

Cϵ

for constant

C >

0

.

For any x ∈ [ cϵ, M ] , we have g ( x ) ≥ min( g ( cϵ ) , g ( M )) . Here we let M > 0 such that ∫ ∞ M p ( x ) dx = ϵ . Then we can lower bound g ( cϵ ) as follows.

If M ≤ 1 , we know | b | < M ≤ 1 . Then we have g ( b ) ≥ g ( cϵ ) .

If M ≥ 1 + ϵ , we can lower bound g ( M ) as

Similarly we will get g ( b ) > min( ψcϵ 2 (1 -c 2 ϵ 2 ) , ϵ 3 +2 ϵ 2 ) . Finally if 1 < M < 1 + ϵ , there exists M ′ > 0 such that ∫ M ′ M q 1 ( x ) dx = ϵ/ 2 . Here M ′ -M > ϵ/ 2 . Then we have

In this case, we have g ( b ) > min( g ( ϵ ) , g ( M ′ )) > min( ψcϵ 2 (1 -c 2 ϵ 2 ) , ϵ 3 / 8+ ϵ 2 / 2) . Therefore, we can lower bound the second moment of ˆ q as follows.

Lemma 16 (Spectral Gap of Covariance) . If a ≤ 0 < b and b > cϵ for constant c > 0 , then the first and second eigenvalues of the uncentered covariance matrix Σ have the following gap

Furthermore, the top eigenvector corresponds to u .

Proof. Weassume wlog that e 1 = u . Then the marginal distribution of P in e 1 is ˆ q , while the marginal distribution in e i is q i for any 2 ≤ i ≤ d . Since q i is isotropic, for any 2 ≤ i ≤ d , E x ∼ P x 2 i = 1 . By

Lemma 17, we have E x ∼ P x 2 1 > 1 + C ′ ϵ 3 for constant C ′ > 0 . Let g ( v ) := E x ∼ P v ⊤ xx ⊤ v v ⊤ v . Then we have g ( e 1 ) > 1 + C ′ ϵ 3 , while g ( e i ) = 1 , ∀ 2 ≤ i ≤ d . Then for any unit vector v = ∑ d i =1 β i e i satisfying ∑ d i =1 β 2 i = 1 , we have

Then we know the top eigenvalue of Σ is λ 1 > 1+ Cϵ 3 . Furthermore, the top eigenvector corresponds to e 1 . Similarly, the second eigenvalue of Σ is λ 2 = max v : v ⊥ e 1 g ( v ) = g ( e i ) = 1 , 2 ≤ i ≤ d . This implies that λ 1 -λ 2 > Cϵ 3 λ 1 for constant C > 0 .

C General Case: Affine Product Distribution with ϵ -Margin

In this section, we first prove the main theorem in Section C.1 using the quantitative lemmas, and then prove all the lemmas. We begin with proving the two qualitative lemmas (Lemma 1 and Lemma 2) in Section C.2, and then prove the quantitative lemmas (Lemma 4 and Lemma 5) in the remaining section. For the quantitative part, we first consider the asymmetric case where | a + b | ≥ ϵ 5 . In this case, contrastive mean leads to recovering u , as elaborated in Section C.3. Secondly, we consider the symmetric case characterized by a + b = 0 , addressed in Section C.4. We show that we can recover u by calculating the top eigenvector of the re-weighted covariance matrix. Finally we extend the result to near-symmetric case where | a + b | < ϵ 5 in Section C.5.

Recall that we are given data x (1) , · · · , x ( N ) drawn from the affine product distribution with ϵ -margin ̂ P . Algorithm 1 first makes the data isotropic. Denote y (1) , · · · , y ( N ) as the corresponding isotropicized data. Then each y ( j ) is an independent and identically distributed variable drawn from P = ˆ q ⊗ q ⊗··· q . Since we compute the re-weighted moments on y ( j ) in the algorithm, we analyze the moments of P directly.

Recall in Definition 1 that q is the symmetric one-dimensional isotropic logconcave density function, and ˜ q is the density obtained by restricting q to R \ [ a, b ] for some unknown a < b . Denote µ 1 , σ 2 1 as the mean and variance of ˜ q . ˆ q is the density obtained after making ˜ q isotropic, with support R \ [ a ′ , b ′ ] , where a ′ = a -µ 1 σ 1 , b ′ = b -µ 1 σ 1 . The density ˆ q on its support is

We denote the standard basis of R d by { e 1 , · · · , e d } , and assume wlog that e 1 = u is the (unknown) normal vector to the band. We write x i := ⟨ x, e i ⟩ as x 's i -th coordinate. We assume in our proof that | b | > | a | . If this condition is not met, we can redefine our interval by setting a ′ = -b and b ′ = -a . The proof can then be applied considering the distribution is restricted to { x ∈ R d : u ⊤ x ≤ a ′ or u ⊤ x ≥ b ′ } . For a vector x ∈ R d , we use ∥ x ∥ to denote its l 2 norm. For a matrix A ∈ R m × n , we denote its operator norm as ∥ A ∥ op.

C.1 Proof of Theorem 1

Armed with two quantitative lemmas and the Davis-Kahan Theorem, we are now prepared to prove the main theorem. We state the formal version of Theorem 1 as follows.

Theorem 7 (Sample Complexity for Affine Product Distribution with ϵ -Margin) . Algorithm 1 with N = ˜ O ( d 6 ϵ -318 δ -2 ξ -1 ) samples learns the target affine product distribution with ϵ -margin to within TV distance δ with probability 1 -ξ .

Proof. Firstly we consider the case when | a + b | ≥ ϵ 5 . Denote α ∗ = argmax α {| ( µ α 1 ) 1 | , | ( µ α 2 ) 1 |} , and µ α = µ α ∗ . By Lemma 4, | ( µ α ) 1 | ≥ C 1 ϵ 159 /d 2 . Since for any negative α , for any 1 ≤ i ≤ d ,

By Chebyshev's Inequality, the re-weighted sample mean ˜ µ = 1 N ∑ N j =1 e α ∗ ∥ y ( j ) ∥ 2 y ( j ) satisfies

Let the sample size N 1 = 4 d 6 C 2 1 ϵ 318 δ 2 ξ = O ( Cd 6 ϵ -318 δ -2 ξ -1 ) , and we have

So we have

This indicates that with probability 1 -ξ , the re-weighted mean can output the vector ˜ µ that is within angle δ to the vector e 1 .

Secondly, for the case when a and b are near-symmetric. Denote Σ as the re-weighted covariance matrix with eigenvalues λ i and ˜ Σ as the empirical re-weighted covariance matrix with eigenvector v . By Lemma 5, λ 1 -λ 2 > C 2 ϵ 3 λ 1 . By Lemma 6, with sample size N 2 = ˜ O ( dϵ -6 δ -2 ξ -1 ) , with probability 1 -ξ ,

So given N = max( N 1 , N 2 ) = ˜ O ( d 6 ϵ 318 δ -2 ξ -1 ) , the algorithm learns the distribution P with probability 1 -ξ .

C.2 Proofs of Qualitative Bounds

We present proofs of two qualitative lemmas: the contrastive mean (Lemma 1) and the contrastive covariance (Lemma 2). Their quantitative counterparts can be found in Section C.3 and Section C.5. To establish the contrastive mean, we invoke Descartes' Rule of Signs. For the proof concerning contrastive covariance, we introduce a novel monotonicity property on the moment ratio, as described in Lemma 3. We include the complete proof within this section.

+

#

> -

+

#

≤ -

)

%

#′

&′

)

%

-#′

&′

)

#′

#′

%

&′

-#′

"

!(-))

"

!())

)

#′

%

&′

"

!(-))

"

!())

Contrastive Mean. We can write the contrastive mean as a linear combination of exponential functions of α . By Descartes' rule of signs, the number of zeros of this function is at most two. Since α = 0 is one root and corresponds to mean zero, there is at most one nonzero root. And thus we have that for any two distinct nonzero α 's, at least one of them achieves nonzero contrastive mean.

Lemma 1 (Contrastive Mean) . If | a + b | > 0 , then for any two distinct nonzero α 1 , α 2 ∈ R , at least one of the corresponding re-weighted means is nonzero, i.e.,

Proof. | b | > | a | implies that µ 1 < 0 . For any x ≥ 0 , we have

Since P is a product distribution, we have

Then we rearrange F ( α ) by combining e αx 2 as in Figure C.1.

If a ′ ≤ 0 , we rewrite F ( α ) as

We treat F ( α ) as the integral of a ( x ) e αx 2 for x ≥ -a ′ . Since ˆ q ( x ) -ˆ q ( -x ) > 0 for x > b ′ , we have a ( x ) > 0 for x ∈ [ -a ′ , b ′ ) and a ( x ) < 0 for x > b ′ . In other words, for increasing x , the sign of a ( x ) only changes once. By Theorem 3, F ( α ) = 0 has at most one root.

If a ′ > 0 , we arrange F ( α ) in the same way and get

Similarly, we treat F ( α ) as the integral of a ( x ) e αx 2 for x ≥ 0 . For increasing x , the sign of a ( x ) changes twice. By Descartes' rule of signs, F ( α ) = 0 has at most two roots. In addition, we know F (0) = E P x 1 = 0 by definition of P . So α = 0 is one root of F ( α ) = 0 . So there is at most one nonzero root of F ( α ) = 0 . In other words, for any two distinct nonzero α 1 , α 2 , at least one of F ( α 1 ) , F ( α 2 ) is nonzero. This implies that

Moment Ratio. To prove Lemma 2, we develop a new monotonicity property of the moment ratio of logconcave distributions. Moment ratio is specifically defined as the ratio of the fourth moment to the square of the second moment of truncated versions of the distribution. This measurement essentially reflects the uncentered kurtosis of the distribution. The formal definition is detailed in Definition 5.

Definition 4 (One-side t -restriction distribution) . Let q be a distribution in one dimension with nonnegative support. For any t ≥ 0 , define q t as the one-side t -restriction distribution on q obtained by restricting q to [ t, ∞ ) .

Definition 5 (Moment Ratio) . Let q be a distribution in one dimension with nonnegative support. For any t ≥ 0 , define q 's moment ratio as a function of t , given by

We will prove the monotonicity of the moment ratio (Lemma 3) by reducing general logconcave distributions to exponential distributions. The monotonicity of the moment ratio for exponential distribution is detailed in Lemma 18.

Lemma18 (Monotonicity of Moment Ratio of Exponential Distribution) . Define h ( x ) = βe -γx , x ≥ 0 , β, γ > 0 . Denote N k ( t ) = ∫ ∞ t x k h ( x ) dx . Then for any t ≥ 0 , we have

We denote

By calculation, we have

Proof. By calculation, we have

Then we can plug them and get

Next, we will prove the monotonicity of moment ratio for logconcave distributions.

Lemma 3 (Monotonicity of Moment Ratio) . Let q be a logconcave distribution in one dimension with nonnegative support. For any t ≥ 0 , let q t be the distribution obtained by restricting q to [ t, ∞ ) . Then the moment ratio of q , defined as mr q ( t ) := var q t ( X 2 ) ( E q t X 2 ) 2 , is strictly decreasing with t .

Proof. Denote M k ( t ) = ∫ ∞ t x k q ( x ) dx . By Definition 5,

Next we will show that mr ′ ( t ) < 0 , ∀ t ≥ 0 . By taking the derivative,

Define H ( t ) = t 4 M 0 ( t ) M 2 ( t )+ M 4 ( t ) M 2 ( t ) -2 t 2 M 0 ( t ) M 4 ( t ) . We will show that H ( t ) > 0 , ∀ t ≥ 0 . Clearly H (0) = M 4 (0) M 2 (0) > 0 , So we only consider t > 0 in the following proof.

Let h ( x ) = βe -γx be an exponential function ( β, γ > 0 ) such that

Then we have

By the logconcavity of q , the graph of h intersects with the graph of q at exactly two points u ′ < v , where v > 0 . Also we have h ( x ) ≤ q ( x ) at the interval [ u ′ , v ] and h ( x ) > q ( x ) outside the interval.

Let u = max { 0 , u ′ } . So for x ≥ 0 , ( x -u )( x -v ) has the same sign as h ( x ) -q ( x ) . Since t ≥ 0 , we have

Expanding and we get

This shows that N 4 ( t ) ≥ M 4 ( t ) . To show that H ( t ) > 0 , we consider two cases.

Firstly if M 2 ( t ) -2 t 2 ≥ 0 , we have

Secondly if M 2 ( t ) -2 t 2 < 0 , by calculation of the exponential function's moments (Lemma 18), we have

Then we have

The equality holds if and only if M 4 ( t ) = N 4 ( t ) . This implies that

Combining both cases, mr ′ ( t ) < 0 , ∀ t ≥ 0 , which implies that the moment ratio of q is strictly decreasing with respect to t .

Contrastive Covariance. View the spectral gap of the re-weighted covariance, denoted as λ 1 ( ˜ Σ) -λ 2 ( ˜ Σ) , as a function of α , which we denote as S ( α ) . By calculation, S (0) = 0 and S ′ (0) is proportional to mr ( b ) -mr (0) , which is negative by the monotonicity property of moment ratio. Then we can prove Lemma 2 using Taylor expansion.

Lemma 2 (Contrastive Covariance) . If a + b = 0 , then there exists an α < 0 , such that (1) there is a positive gap between the top two eigenvalues of the re-weighted uncentered covariance matrix ˜ Σ = E x ∼ P e α ∥ x ∥ 2 ( xx ⊤ ) . That is, λ 1 ( ˜ Σ) > λ 2 ( ˜ Σ) . (2) The top eigenvector of ˜ Σ is u .

Proof. Denote M k ( t ) = ∫ ∞ t x k q ( x ) dx . The variance σ 2 1 of q restricted to R \ [ -b, b ] is

Since ˆ q is isotropic, the density on the support R \ [ -b/σ 1 , b/σ 1 ] is

Let

Since q and ˆ q are both isotropic, S (0) = E x ∼ ˆ q x 2 -E x ∼ q x 2 = 0 . Then,

The last step is because the distribution q is isotropic. By Lemma 3, we know S ′ (0) = mr ′ (0) < 0 . On the other hand, ∀ α ≤ 0 , S ′′ ( α ) can be bounded.

By Taylor expansion, we know there exists α < 0 such that

Then we have for 2 ≤ j ≤ d ,

For any v ∈ R d , define ϕ ( v ) as

Substituting v with e 1 and e j , 2 ≤ j ≤ d , we have

This implies that for 2 ≤ j ≤ d ,

For any vector v = ∑ d i =1 γ i e i , we have

This shows that the top eigenvalue of ˜ Σ is λ 1 ( ˜ Σ) = max v ϕ ( v ) = ϕ ( e 1 ) . Similarly, λ 2 ( ˜ Σ) = max v : v ⊥ e 1 ϕ ( v ) = ϕ ( e j ) , 2 ≤ j ≤ d . Therefore, λ 1 ( ˜ Σ) > λ 2 ( ˜ Σ) and the top eigenvector is e 1 , which is essentially u .

C.3 Quantitative Bounds for Contrastive Mean

We will prove Lemma 4 in this section. Here we consider the case when | a + b | ≥ ϵ 5 . We compute the contrastive mean of P given α < 0 as E x ∼ P e αx 2 x using two different α 's.

Definition 6. We define F ( α ) as re-weighted mean for the one-dimensional distribution ˆ q .

Since P is isotropic, F (0) = E x ∼ ˆ q x = 0 .

̸

To prove Lemma 4, we need to show that for given α 1 , α 2 , the maximum of | F ( α 1 ) | , | F ( α 2 ) | exceeds a certain positive threshold. We follow the same idea of bounding the number of roots of F ( α ) as in the qualitative lemma (Lemma 1). By taking the derivative of F ( α ) , we can show that either F ′ (0) = 0 or F ′′ (0) = 0 . Then by Taylor expansion, we can choose two distinct α 's (near zero) so that one of the corresponding contrastive means is bounded away from zero.

̸

In the process of proving the quantitative bounds, similar to our approach with qualitative bounds, we must consider two distinct scenarios based on the sign of a ′ , as illustrated in Figure 4.1.

In the case where a ′ is negative, Lemma 20 asserts that the first derivative of F at zero, F ′ (0) , is always positive.

Conversely, when a ′ is nonnegative, Lemma 23 reveals an essential characteristic of the function F ( α ) : it's not possible for both F ′ (0) and F ′′ (0) to be zero at the same time.

Lemma 24 provides upper bounds for the derivatives of F ( α ) . These upper bounds are crucial as they help in managing the extra terms that emerge during the Taylor expansion of F ( α ) .

The section concludes with the proof of Lemma 4, which is the quantitative lemma for the contrastive mean.

We start with Lemma 19 showing that | µ 1 | is away from zero provided that | a + b | is also different from zero.

Lemma 19 (Lower Bound of | µ 1 | ) . If | a + b | ≥ ϵ s for s ≥ 2 , then | µ 1 | ≥ ϵ s / 2 e .

Proof. Firstly, let's consider the case when a ≤ 0 . By Lemma 9, we know q ( x ) is upper bounded by 1 . Since | a + b | ≥ ϵ s and q is logconcave, by Lemma 8, the mean of the density restricted q in [ -a, b ] satisfies µ [ -a,b ] ≥ 1 /e . Then

Since | a | < | b | , we have

So we have

Secondly, when a > 0 , since b -a > ϵ ,

Lemma 20 (Derivative of F (0) when a ′ < 0 ) . If | a + b | ≥ ϵ s for s ≥ 2 and a ′ < 0 , then F ′ (0) > ϵ 3 s +3 . 5 / 2 .

Proof. We rearrange F ( α ) by combining terms with same e αx 2 as in Figure 4.1, and get

Define r ( x ) = { -ˆ q ( -x ) x ∈ [ -a ′ , b ′ ] ˆ q ( x ) -ˆ q ( -x ) x ∈ [ b ′ , ∞ ) . Then we have

By calculating the derivative of F ( α ) , we have

Since r ( x ) is nonnegative for x ≥ b ′ and negative otherwise, then for any x ≥ -a ′ , we have

Since F (0) = 0 , we have

By calculation,

Choose t 0 ∈ [ -a, b ] such that ∫ t 0 -a q ( x ) dx = ∫ b t 0 q ( x ) dx . Since q ( x ) is bounded by 1 by Lemma 9, we have

On the other hand, similar to the proof of Lemma 19, we have

So we have

By definition, we have

Applying Lemma 19, we know | µ 1 | > ϵ s / 2 e . Using these results to estimate F ′ (0) , we get

Lemma 21. If a ≥ µ 1 , µ 1 ≤ 0 , then we have ∫ b a -2 µ 1 xq ( x ) dx ≥ | µ 1 | ∫ ∞ b q ( x ) dx .

Proof. Firstly we will show that ∫ a 2 µ 1 -a ( x -µ 1 ) q ( x ) dx ≥ 0 . For x ∈ [ µ 1 , a ] , we have (2 µ 1 -x ) 2 -x 2 = 4 µ 1 ( µ 1 -x ) ≥ 0 . Since q ( x ) is symmetric and uni-modal, we have q (2 µ 1 -x ) ≤ q ( x ) , ∀ x ∈ [ µ 1 , a ] .

And then, we have

Since µ 1 is the mean of the distribution ˜ q . We have

Then we know

Then we have

This derives that

Lemma 22 (Second Derivative of F (0) when a ′ ≥ 0 ) . If a ′ ≥ 0 , we define the following functions,

Then we have

Proof. We rearrange F ( α ) by combining terms with same e αx 2 as in Figure 4.1, and get

By the definition of r ( x ) , we naturally have

Then we can calculate its first and second derivative as follows.

By the definition of H ( α ) , we have

Lemma 23 (First and Second Derivatives of F (0) when a ′ ≥ 0 ) . If | a + b | ≥ ϵ s for s ≥ 2 and a ′ ≥ 0 , then we have either F ′ (0) < -C 2 ϵ 6 s +6 . 5 log 4 (1 /ϵ ) or F ′′ (0) > C 3 ϵ 6 s +4 . 5 log 4 (1 /ϵ ) for constants C 2 , C 3 > 0 .

Proof. We prove the lemma by showing that H (0) > C 1 ϵ 6 s +4 . 5 log 5 (1 /ϵ ) . We calculate H (0) as follows.

Denote ρ := ∫ b a -2 µ 1 q ( x ) dx , by Lemma 21 and the bound of b , we know

Choose t 1 < t 2 such that ∫ t 1 a -2 µ 1 q ( x ) dx = ∫ t 2 t 1 q ( x ) dx = ∫ b t 2 q ( x ) dx . Since q ( x ) is upper bounded by 1 by Lemma 9, we have

Using this, we can bound

√

We have shown that σ 1 ≤ 1 / 1 -∫ b a q ( x ) dx . Combining all results, we can compute H (0) as

Given F (0) = 0 , by Lemma 22,

Since the distribution is symmetric, we know

So we know 0 ≤ a ′ < b ′ = b -2 µ 1 < 2 . 5 /ϵ , thus ( a ′ 2 + b ′ 2 ) 2 < 25 / (2 ϵ 2 ) . Thus we have either F ′ (0) < -C 2 ϵ 6 s +6 . 5 log 4 (1 /ϵ ) or F ′′ (0) > C 3 ϵ 6 s +4 . 5 log 4 (1 /ϵ ) for constants C 2 , C 3 > 0 .

Lemma 24 (Upper Bound of F 's derivatives) . For α ≤ 0 , the derivatives of F ( α ) are bounded as

Proof. Define M k = E x ∼ q | x | k . By Cauchy-Schwarz Inequality, M 1 ≤ √ M 2 M 0 = 1 . By Lemma 12,

Also we have proved that | µ 1 | ≤ 1 /ϵ . By definition of F ( α ) , we calculate its first derivative as follows.

Similarly, we calculate its second and third derivatives as follows.

Now we are ready to prove Lemma 4.

Lemma 4 (Quantitative Gap of Contrastive Mean) . Suppose that | a + b | ≥ ϵ 5 . Then, for α 1 = -c 1 ϵ 82 /d, α 2 = -c 2 ϵ 42 /d , the re-weighted mean of P , denoted as µ α 1 and µ α 2 , satisfies

Proof of Lemma 4. For any 2 ≤ k ≤ d , for any α , by symmetry of q , the contrastive mean is

Next we will consider E x ∼ P e α ∥ x ∥ 2 x 1 . For any x ≥ 0 , we have

Since P is a product distribution, we have

We will consider two cases depending on whether a ′ ≥ 0 . See Figure 4.1.

Firstly, if a ′ ≤ 0 . We use α 2 in this case. By Lemma 20, F ′ (0) > ϵ 18 . 5 / 2 . By Taylor expansion, there exists α 2 < η 0 < 0 such that

By Lemma 24, we know F ′′ ( η 0 ) ≤ C 5 /ϵ 5 . Since F (0) = 0 , we know for α 2 = -c 2 ϵ 42 /d ,

Then we consider the case when a ′ > 0 . By Lemma 23, we have either F ′ (0) < -C 2 ϵ 36 . 5 log 4 (1 /ϵ ) or F ′′ (0) > C 3 ϵ 34 . 5 log 4 (1 /ϵ ) . Here we consider three cases with respect to F ′ (0) .

Case 1 : F ′ (0) < -C 2 ϵ 36 . 5 ln 4 (1 /ϵ ) . We use α 2 in this case. By Taylor expansion, there exists η 1 such that α 2 < η 1 < 0 and

By Lemma 24, we know | F ′′ ( η 1 ) | ≤ C 5 /ϵ 5 . By choosing α 2 = -c 2 ϵ 42 /d ,

Case 2 : -C 2 ϵ 36 . 5 ln 4 (1 /ϵ ) ≤ F ′ (0) ≤ c s ϵ 77 d with some constant c s > 0 . We use α 2 in this case. By Lemma 23, we know F ′′ (0) > C 3 ϵ 34 . 5 / ln 4 (1 /ϵ ) . Then there exists η 2 satisfying α 2 < η 2 < 0 and

By Lemma 24, we know | F ′′′ ( η 2 ) | ≤ C 6 /ϵ 7 . Thus by choosing α 2 = -c 2 ϵ 42 /d ,

Case 3 : F ′ (0) > c s ϵ 77 d . We use α 1 in this case. Then there exists η 3 satisfying α 1 < η 3 < 0 and

By Lemma 24, we know | F ′′ ( η 3 ) | ≤ C 5 /ϵ 5 . For α 1 = -c 1 ϵ 82 /d , we have

Then we know for all cases, there exists a constant C ′ = min( C 7 , C 8 , C 9 , C 10 ) such that

Finally we will lower bound ( E x ∼ q e αx 2 ) d -1 as follows.

Let C = C ′ /e , and we will get

C.4 Quantitative Bounds for Contrastive Covariance: Symmetric Case

Before addressing Lemma 5 which is applicable in the scenario where | a + b | < ϵ 5 , we first demonstrate that contrastive covariance works for the case where the removed band [ a, b ] is symmetric around the origin. That is, a = -b . In such cases, we aim to establish that there's a noticeable difference between the top two eigenvalues ( λ 1 and λ 2 ) of the contrastive covariance matrix ˜ Σ , stated in Lemma 25. We will then extend the lemma to the near-symmetric scenario in Section C.5.

Lemma 25 (Quantitative Spectral Gap of Contrastive Covariance - Symmetric Case) . Suppose a + b = 0 . Choose α 3 = -c 3 ϵ 2 for some constant c 3 > 0 . Then, for an absolute constant C , the top two eigenvalues λ 1 ≥ λ 2 of the corresponding re-weighted covariance of P satisfy

We recall the definition of moment ratio as in Definition 5.

For simplicity, in the remaining section, we'll use mr ( t ) as a shorthand notation for this moment ratio. Just as in the proof of the qualitative bound in Section C.2, the difference between the first and second eigenvalues (the spectral gap) of the re-weighted covariance matrix, denoted as λ 1 -λ 2 , is proportional to the difference in the moment ratio of the distribution q at 0 and b , denoted as mr (0) -mr ( b ) . To prove the quantitative result Lemma 25, our proof strategy involves several steps.

Proving monotonicity of Moment Ratio mr ( t ) . This property is stated in Lemma 3 and its proof is presented in Section C.2. The proof involves reducing the case of general logconcave distributions to that of exponential distributions.

Establishing a positive spectral gap for small b . With Lemma 26, we focus on illustrating that for values of b which are relatively small (less than a certain constant), there is a guaranteed positive gap mr (0) -mr ( b ) .

Generalizing to any b satisfying ∫ b -b q ( x ) dx ≥ ϵ in Lemma 29. The lemma will directly imply the quantitative result lemma for the symmetric case (Lemma 25).

Having demonstrated the monotonicity of the moment ratio in Section C.2, we now begin by illustrating the positive gap, denoted as mr (0) -mr ( b ) , for small b in Lemma 26. The proof relies on the properties of moments derived by the unimodality of the distribution (Lemma 27, Lemma 28).

Lemma 26 (Gap for small t ) . For t > 0 such that ∫ t 0 q ( x ) dx ≤ 0 . 4 , we have the following gap

Proof. Denote ∫ t 0 q ( x ) dx = ν, ∫ t 0 x 2 q ( x ) dx = ω . Then we have

Fix q ( t ) and ν , we apply Lemma 27 and have

Since q ( x ) is monotonically decreasing, for any x ≤ t, q ( x ) ≥ q ( t ) . So we have the constraint that

Plug into the previous inequality and we get

In addition, by fixing v and t , we apply Lemma 28 and get

So we know

By calculation, we will get

Let T ( ν ) = -8 ν 3 +24 ν 2 -30 ν +9 , then we know its derivative is

So T ( ν ) is monotonically decreasing. Since ν < 0 . 4 , we know

Plugging in and we will get

Finally by Cauchy-Schwarz Inequality, we have M 0 ( t ) M 4 ( t ) ≥ M 2 2 ( t ) .

Lemma 27. Let t, r ≥ 0 , 0 < s ≤ 1 . Define P = { p ( x ) : [ t, ∞ ) → [0 , 1) , logconcave , p ′ ( x ) ≤ 0 , ∫ ∞ t p ( x ) dx = s, p ( t ) = r } . Then we have

Proof. For any p ∈ P , we denote M 2 ( p ) = ∫ ∞ t x 2 p ( x ) dx . Define

Clearly u ∈ P . We will show that u ( x ) = argmin p ∈ P M 2 ( p ) . For any p ∈ P , we have ∫ ∞ t u ( x ) dx = ∫ ∞ t p ( x ) dx , u ( t ) = p ( t ) and p ′ ( x ) ≤ 0 . So we know the graph of u and p intersects at points t and t + s/r , where u ( x ) ≥ p ( x ) in the interval [ t, t + s/r ] and u ( x ) < p ( x ) outside the interval. So we know

Since for any x ∈ [ t, t + s/r ] , any y ∈ [ t + s/r, ∞ ) , we have x ≤ y . So we have

This shows that

By calculating M 2 ( u ) , we have

Lemma 28. Let t ≥ 0 , 0 < s ≤ 1 . Define P = { p ( x ) : [0 , t ] → [0 , 1) , logconcave, p ′ ( x ) ≤ 0 , ∫ t 0 p ( x ) dx = s } . Then we have

Proof. For any p ∈ P , we denote M 2 ( p ) = ∫ t 0 x 2 p ( x ) dx . Define u ( x ) = s t · 1 x ∈ [0 ,t ] . Clearly, u ∈ P . Then for any p ∈ P , because it is monotonically decreasing and ∫ t 0 p ( x ) dx = ∫ t 0 u ( x ) dx , the graphs of p and u intersect at point l ∈ [0 , t ] . Also p ( x ) ≥ u ( x ) for x ∈ [0 , l ] and p ( x ) ≤ u ( x ) for x ∈ [ l, t ] . Since for any x ∈ [0 , l ] and any y ∈ [ l, t ] , we have x ≤ y . So we know

By calculating M 2 ( u ) , we have

Using Lemma 3 and Lemma 26, we will show the positive gap mr (0) -mr ( b ) for any b satisfying ∫ b -b q ( x ) dx ≥ ϵ as in Lemma 29.

Lemma 29 (Gap for Log-concave Distribution) . Let 0 < ϵ < 0 . 1 , let b > 0 satisfying ∫ b 0 q ( x ) dx ≥ ϵ/ 2 . Then we have for mr (0) -mr ( b ) ≥ ϵ/ 200 .

Proof. Let b 0 such that ∫ b 0 0 q ( x ) dx = ϵ/ 2 . By Lemma 26, we know

By Lemma 3, we know mr ′ ( t ) ≤ 0 , ∀ t ≥ 0 . So for any b > 0 such that ∫ b 0 q ( x ) dx ≥ ϵ/ 2 , mr ( b ) ≤ mr ( a 0 ) . Therefore,

Before moving on to the proof of the quantitative lemma, we will first present a helper lemma that can be directly applied.

Lemma 30. Define

For a given α = -cϵ s with s ≥ 2 and a certain positive constant c , it can be established that

Proof. We show the lower bound of S ( α ) using Taylor expansion.

Firstly, since q and ˆ q are both isotropic,

Secondly, we will lower bound | S ′ (0) | using the monotonicity of moment ratio. The variance σ 2 1 of q restricted to R \ [ -b, b ] is

Since ˆ q is isotropic, the density on the support R \ [ -b/σ 1 , b/σ 1 ] is

By calculation, we have

Then we can compute S ′ (0) as

The last step is because the q is isotropic. By Lemma 29, mr (0) -mr ( b ) ≥ ϵ/ 200 . This indicates that S ′ (0) ≤ -ϵ/ 200

Next, we can upper bound S ′′ ( α ) for any α ≤ 0 as

By Cauchy-Schwarz Inequality, M 1 (0) ≤ √ M 2 (0) M 0 (0) = 1 / 2 . By Lemma 12,

Since M 0 ( b ) ≥ ϵ , for some positive constant c sec ,

By Taylor expansion, we know for α 3 = -cϵ s , s ≥ 2 , c = 1 / (101 c sec ) , there exists α ′ ∈ [ α, 0] such that for some constant C > 0 ,

Now we are ready to prove the contrastive covariance lemma (Lemma 25).

Proof of Lemma 25. Define

Then for 2 ≤ j ≤ d ,

Since α 3 < 0 , we have

By Lemma 30, α 3 = c 3 ϵ 2 implies that S ( α 3 ) > Cϵ 3 . So we have

Finally we will show that the first eigenvector corresponds to e 1 . For any v ∈ R , define ϕ ( v ) as

Then we know for 2 ≤ j ≤ d ,

For any vector v = ∑ d i =1 γ i e i , we have

̸

This shows that the top eigenvalue of ˜ Σ is λ 1 = max v ϕ ( v ) = ϕ ( e 1 ) . In other word, the top eigenvector is e 1 , which is essentially u . Similarly the second eigenvalue of ˜ Σ is λ 2 = max v : v ⊥ e 1 ϕ ( v ) = ϕ ( e j ) , 2 ≤ j ≤ d . So we get

C.5 Quantitative Bounds for Contrastive Covariance: Near-Symmetric Case

We have shown the result for symmetric case a + b = 0 in Section C.4. Here we will show that we can extend the contrastive covariance lemma (Lemma 25) to the near-symmetric case, where | a + b | < ϵ 5 . In this section, we will present the proof of Lemma 5, which addresses the nearly symmetric case quantitatively. The proof idea is to approximate the re-weighted covariance of the distribution with margin [ a, b ] , by comparing it to the same distribution truncated with the symmetric interval [ -b, b ] . This enables us to generalize the result from the symmetric scenario to the near-symmetric scenario.

̸

̸

Recall that ˜ q is the distribution obtained by restricting q to the set R \ [ a, b ] . We denote ˜ r as the distribution that is obtained by restricting q to the set R \ [ -b, b ] , and ˆ r as the isotropized distribution of ˜ r . Let σ 2 2 be the variance of ˜ r .

To approximate the characteristics of ˆ q using those of ˆ r , we undertake the subsequent steps.

Assessing the mean. We illustrate that the mean of ˜ q is adequately small in Lemma 31.

Variance approximation. We approximate the variance of ˆ q by using the variance of ˆ r , as elaborated in Lemma 32.

Re-weighted second moment. We use the re-weighted second moment of ˆ r to approximate the corresponding moment in ˆ q . The details are provided in Lemma 34.

Re-weighted zeroth moment. We use the re-weighted zero moment of ˆ r to approximate the corresponding moment in ˆ q , which is shown in Lemma 35.

Lemma 31. For an integer s ≥ 1 , if | a + b | < ϵ s +1 , then

the mean of ˜ q , | µ 1 | < ϵ s ln(1 /ϵ ) ;

a < 0 .

Proof. We first consider the case when a < 0 . By Lemma 8, we have

By the tail bound of logconcave distributions (Lemma 11),

Then we have

Next for a ≥ 0 , we can bound b by ϵ s +1 / 2 because | b | ≥ | a | . This implies that b -a ≤ ϵ s +1 / 2 , which leads to a contradiction that ∫ b a q ( x ) dx ≥ ϵ . Therefore, a can only be negative in this scenario.

Lemma 32. For | µ 1 | ≤ r with r < 1 / 6 , we can bound the variance as follows.

Proof. By Lemma 31, we know a < 0 . We can calculate the variance as

On one hand,

So we have

On the other hand, since | µ 1 | ≤ r , by Lemma 8,

So we have

Lemma 33. The variance σ 2 2 is monotonically increasing with respect to b . Furthermore, σ 2 2 ≥ 1 . Proof. By taking the derivative,

So for b > 0 ,

Lemma 34 (Approximation for Re-weighted Second Moment) . For | a + b | ≤ ϵ 5 , by choosing α = -cϵ 2 , then for some constant C ′ > 0 , we have the following inequalities.

Proof. By Lemma 31, we can bound | µ 1 | as

We begin by showing that ∫ ∞ b e αy 2 /σ 2 2 y 2 q ( y ) dy is close to ∫ ∞ b e αy 2 /σ 2 1 y 2 q ( y ) dy . By Lemma 32,

Since α < 0 , αy 2 /σ 2 1 < αy 2 /σ 2 2 for y > 0 . This implies

On the other hand,

On the other hand,

The last inequality is implied by Lemma 12. Furthermore, by Lemma 33, σ 2 2 ≥ 1 . So there exists a constant c 1 > 0 such that

This also applies for the integral from -a to ∞ . To be specific,

Next we will show that ∫ ∞ b e αy 2 /σ 2 1 y 2 q ( y ) dy and ∫ ∞ b e α ( y -µ 1 ) 2 /σ 2 1 y 2 q ( y ) dy are close to each other. Since µ 1 < 0 , α < 0 , we derive that αy 2 > α ( y -µ 1 ) 2 . This implies

The last inequality is implied by Lemma 12 and Lemma 33 . Combining two inequalities, we get

Similarly, we will show the approximation inequality for ∫ ∞ -a e α ( y + µ 1 ) 2 /σ 2 1 y 2 q ( y ) dy . We will decompose the integral by the summation of the integral on [ -a, 6 ln(1 /ϵ )] and (6 ln(1 /ϵ ) , ∞ ) respectively. For the first part of the integral,

∣ ∣ ∣ ∣ ∣ ∫ 6 ln(1 /ϵ ) -a e α ( y + µ 1 ) 2 /σ 2 1 y 2 q ( y ) dy -∫ 6 ln(1 /ϵ ) -a e αy 2 /σ 2 1 y 2 q ( y ) dy ∣ ∣ ∣ ∣ ∣ = ∣ ∣ ∣ ∣ ∣ ∫ 6 ln(1 /ϵ ) -a e αy 2 /σ 2 1 ( e αµ 1 σ 2 1 (2 y + µ 1 ) -1 ) y 2 q ( y ) dy ∣ ∣ ∣ ∣ ∣ ≤ ∫ -µ 1 / 2 -a e αy 2 /σ 2 1 ( 1 -e αµ 1 σ 2 1 (2 y + µ 1 ) ) y 2 q ( y ) dy + ∫ 6 ln(1 /ϵ ) -µ 1 / 2 e αy 2 /σ 2 1 ( e αµ 1 σ 2 1 (2 y + µ 1 ) -1 ) y 2 q ( y ) dy The first term can be bounded using e -t ≥ 1 -t . ∫ -µ 1 / 2 -a e αy 2 /σ 2 1 ( 1 -e αµ 1 σ 2 1 (2 y + µ 1 ) ) y 2 q ( y ) dy ≤ ∫ -µ 1 / 2 -a αµ 1 σ 2 1 ( -2 y -µ 1 ) y 2 q ( y ) dy ≤ ∫ -µ 1 / 2 -a | α || µ 1 | 2 y 2 q ( y ) dy ≤ ϵ 8

The second term can be bounded using the upper limit of the integral. For y ≤ 6 ln(1 /ϵ ) ,

Substituting into the second term, we get

Combining both terms,

For the remaining part of the integral, we can bound using logconcave distribution's upper bound as in Lemma 13. For any t ≥ 3 ,

Then the following holds with some constant c ′ 3 > 0 .

Similarly, we get

With Equations (C.9), (C.10), (C.11), we get

Combining Equations (C.6), (C.7), (C.8), (C.12) , we have

By choosing C ′ = c 1 + c 2 + c 3 , we prove the lemma.

Lemma 35 (Approximated for Re-weighted Zeroth Moment) . By choosing α = -cϵ 2 , for some constant C ′ > 0 , we have the following inequalities.

The proof follows exactly from the proof of Lemma 34 by replacing y 2 with 1 .

Now we are ready to prove Lemma 5.

Lemma 5 (Quantitative Spectral Gap of Contrastive Covariance) . Suppose that | a + b | < ϵ 5 . Choose α 3 = -c 3 ϵ 2 for some constant c 3 > 0 . Then, for an absolute constant C , the top two eigenvalues λ 1 ≥ λ 2 of the corresponding re-weighted covariance of P satisfy

Proof. By calculation, we have

The first term is close to E x ∼ ˆ r e αx 2 x 2 while the second and third terms are close to zero. We first give the bound on the absolute values of last two terms. Since α < 0 ,

Similarly,

By Lemma 32, Lemma 34 and Lemma 35, we have

By combining with the second and third terms, we conclude that for constants c 3 , c 4 > 0 ,

Similarly, we have

Then, we would like to compute the gap between first and second eigenvalues of the re-weighted second moment of ˆ q . We denote Q as the product of ˆ q and d -1 fold of q . For 2 ≤ j ≤ d ,

We define T ( α ) as follows.

Recall that S ( α ) is defined in Lemma 30.

We calculate the difference between T ( α ) and S ( α ) using Equation (C.15) and Equation (C.16).

By Lemma 30, S ( α ) > C ′ ϵ 3 for constant C ′ > 0 . So we know T ( α ) > Cϵ 3 for C > 0 . Then all proof follows as same as the case when a + b = 0 . We write out the proof for completeness. For 2 ≤ j ≤ d ,

Since α 3 < 0 , we have

Also we have shown that T ( α 3 ) > Cϵ 3 . So we have

Finally we will show that the first eigenvector corresponds to e 1 . For any v ∈ R , define ϕ ( v ) as

For any vector v = ∑ d i =1 γ i e i , we have

Then we know for 2 ≤ j ≤ d ,

̸

This shows that the top eigenvalue of ˜ Σ is λ 1 = max v g ( v ) = g ( e 1 ) . In other word, the top eigenvector is e 1 . Similarly the second eigenvalue of ˜ Σ is λ 2 = max v : v ⊥ e 1 ϕ ( v ) = ϕ ( e j ) , 2 ≤ j ≤ d . So we get

̸

̸

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nAssuming that the data is generated from a full-rank affine transformation of a $d$-fold product distribution, one of which is divided into two groups by a halfspace. The authors address the problem of learning the halfspace by weighting the isotropized sample so that either the corresponding sample mean or the top eigenvector of the covariance matrix shifts along the normal direction to the halfspace.\n\nSTRENGTHS:\n- Good presentation of theoretical results, with the proof of the main theorem summarized succinctly.\n- Interesting idea of using Descartes'  Rule of Signs to single out a nonzero mean from weighted samples.\n- Experimental results agree with the theory.\n- In contrast to prior work which focuses on Gaussian data, the proposed method extends to data generated from logconcave distributions.\n\nWEAKNESSES:\n- Real-world applications is still unclear to me. In what situation do we expect the set-up in Definition 1 to arise? The authors list the methods that benefit from the halfspace learning in Line 40-49, but where do these methods apply?\n- Related to the point above, the authors might want to test their method on real-world data.\n- The paper seems to have been reorganized but some notations are left unchanged---see questions below.\n\nQUESTIONS:\n- The purpose of Lemma 1 and Lemma 2 is unclear. From a quick search in the supplement, they are not used to prove the main Theorem. And I personally think that Lemma 4 and Lemma 5 are more informative. Do the maximum of the two expectations shows up in the proof of Lemma 4?\n- What are $\\mu_{\\alpha_1}$ and $\\mu_{\\alpha_2}$ in Lemma 4?\n- What are $a'$ and $b'$ in the Proof of Lemma 1?\n- Bottom of Page 6: \"Since $\\hat{q}(x) - \\hat{q}(-x) \\geq 0$ for $x\\geq b'$.\" looks like an incomplete sentence.\n- Proof Idea of Lemma 4: I do not understand the statement \"$F'(0)$ has at most one root.\" and \"$F\"(0)$ has no root.\"\n- Do the time and sample complexities also depend on the distribution's tail decay rate?\n\nLIMITATIONS:\nThe authors address some challenges of improving the sample complexity bound to linear, and extending the algorithm to more general distributions.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a non-standard approach to the problem of learning halfspaces, where instead of labelled samples, the learner only receives unlabelled ones as well as strong prior information about the unknown distribution. In particular, the marginal distribution is assumed to correspond to some affinely skewed product distribution, with identical components except from one which is truncated outside some interval. The unsupervised distribution learning problem corresponds to a supervised learning problem when the labels are determined by a halfspace in the direction of the truncation.\n\nThe main result shows that when the truncation region is wide enough but not too wide and either symmetric or bounded away from being symmetric, then the direction of truncation can be discovered efficiently from unlabelled samples. The algorithm exploits standard ideas from distribution learning (data normalization, PCA and sample re-weighing), but its proof involves careful application of appropriate tools.\n\nSTRENGTHS:\nThe paper proposes and studies an interesting connection between unsupervised and supervised learning. Moreover, I appreciated the technical work required to prove guarantees about the main algorithm, even though it is based on simple and clean algorithmic ideas.\n\nWEAKNESSES:\nThere are several concerns I have regarding the setup, the impact and the results. In particular:\n\n1. While the model demonstrates an interesting connection between supervised and unsupervised learning, if we focus on either of these aspects, it seems less relevant. For supervised learning, the distributional assumptions are too strong (usually one either makes strong assumptions on the marginal distribution or assumes the existence of margin, but not both). For unsupervised learning, the component we want to discover is too specific (instead of a general property, like matching some Gaussian moments, it is assumed that the component is just a truncation of the distribution of the other components).\n    \n2. On the algorithmic level, the impact seems to be limited. Variants of the algorithm presented have largely appeared in the literature for similar problems.\n    \n3. The main result (Theorem 1) does not seem to be completely accurate. More specifically, there has been no account for the case where the truncation region $[a,b]$ has $|a+b|>0$ but $|a+b|\\approx 0$ (e.g., $|a+b| = 2^{-d}$). While Lemma 4 accounts for the case when $|a+b|$ is large enough, Lemma 5 only accounts for the case $|a+b|=0$. It is not clear what happens in between. In Figure 5.2, it seems that Lemma 5 could be generalized to capture cases when $|a+b|$ is positive but bounded. However, this has not been proven.\n\nOverall, the paper seems to have important unresolved technical issues and also it could highly benefit by relaxing the distributional assumptions. I would encourage the authors to continue working on the paper, but in its current form, it is below the acceptance threshold.\n\n-- The authors fixed an important technical issue in their proof (item 3 above) and I have increased my score (from 3 to 5) to reflect this.\n\nQUESTIONS:\nI would be interested to hear any comments that the authors have regarding the points (1,2 and 3) I made in the weaknesses section. Moreover, I have the following suggestions regarding the presentation.\n\n1. The problem definition (page 2) needs to be clarified further to avoid confusion. In particular, it is not clear what is known and what is unknown. For example, does the learner know $q,a$ and/or $b$? \n\n2. How does the paper [DK'22] compare to your results? On a more general note, I think it would be beneficial to present in section 1.2 some result on NGCA (e.g. [DK'22]) more concretely, as well as some result on supervised learning and emphasize the trade-offs when considering the problem proposed in this paper.\n\n[DK'22]: Diakonikolas, Ilias, and Daniel Kane. \"Non-gaussian component analysis via lattice basis reduction.\" Conference on Learning Theory. PMLR, 2022.\n\n3. Some typos:\n- Line 105: double \"the\".\n- Line 106: \"follows uses\" $\\to$ \"uses\".\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work addresses learning a halfspace with margin from unlabeled samples (which is possible due the combination of margin and distributional assumptions). They impose that the ambient distribution is an affine transformation of the d-fold product of some 1D symmetric log-concave distribution, conditioned on one coordinate lying outside of an interval with mass at least $\\varepsilon$. They provide an algorithm for recovering this distribution up to TV error $\\delta$ with time and sample complexity $poly(1/\\varepsilon,1/\\delta,d)$, based on analyzing the first two moments of a certain exponential reweighting of the empirical distribution.\n\nSTRENGTHS:\n- The paper was generally well-written and organized. The proposed algorithm is easy to implement, and its analysis requires some clever new techniques.\n- I appreciate the initial simplified lemmas; they allow the reader to understand main ideas at a high-level without getting bogged down with the full details.\n- Approaches for this problem based on previous work would seem to require Gaussianity and fail to provide as strong recovery guarantees in polynomial time\n- Experiments validate their algorithm on Gaussian and non-Gaussian data\n\nWEAKNESSES:\n- The notion of margin seems a bit non-standard. Can the authors relate this to the standard notion (or refer to an existing example of this usage)?\n- The proven time/sample complexity, while polynomial, is quite slow\n- While the authors suspect that improved guarantees are possible for the same algorithm, it would be preferable if they could support this conjecture with experiments; the current sample sizes are quite large\n\nMinor Nits:\n- Some inconsistency in Sec 3 for whether $\\hat{q}$ is isotropized or not (it mostly is, but not in the warm up)\n- Inconsistency between single bars and double bars for the Euclidean norm (e.g. Lemma 1 uses both)\n- Inconsistency between $\\mu_i$ and $\\mu_{\\alpha_i}$ in Sec 3\n- Line 214, clause beginning with \"Since\" looks like it should be combined with following sentence\n- Line 217, should be for $x \\geq 0$?\n- Perhaps the rule of signs should be stated in the integral form for which is applied? Also, it would improve readability to mention the u-substitution used to eliminate the square in the exponential\n- Line 225, $F'(0)$ is a scalar, doesn't have a root, same with $F''(0)$ (presumably should just be $\\alpha$)\n- Line 226, doesn't $F''(0) \\neq 0$ always hold?\n- Equation after line 240, would match previous notation to use $\\hat{q}(x)$?\n\nQUESTIONS:\n- The results are stated for non-singular affine transformations. What happens as the transformation approaches singularity? Given that there appears to be no breakdown in the run time bound in this regime, the non-singularity requirement was surprising to me\n\nLIMITATIONS:\nYes, authors are clear about assumptions and potential impacts.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-08"}

comments: SUMMARY:
In this work, the authors study the problem of unsupervised learning a halfspace. In their setting they have a continuous distribution (isotropic-log-concave) and they remove $\geq \epsilon$ mass from an interval $u\cdot x\in[a,b]$, and their goal is to find the half space that separates the two regions. Their idea is basically to reweight the samples and use the first two moments to find the unknown vector.

STRENGTHS:
Well written and easy to follow. The authors provide experiments a proof of concept. This is a nice problem and it was not studied before. Even if the method of attacking this problem is what someone expect, still I found the results intersting.

WEAKNESSES:
1. The distributional assumptions are strong. They consider a product of the same log-concave distribution and also they assume that this distribution is symmetric. 
2. Some steps that are trivial are over-simplified where some other steps that are not easy to follow are not explained, especially in proof of lemma 4.
3. The results are not tight as is needed a high polynomial on $\epsilon$.

QUESTIONS:
545-546: Is this part of the proof ok? The distribution is no longer symmetric after removing the mass.
I think this problem is closed related to learning with truncated samples. Maybe the authors should comment on this/add related work.

typos:
550: I guess you mean that the directions are uncorrelated as after removing the mass and rotating this is not a product distribution but it is true that the directions are uncorrelated.
529: there is a missing number.

LIMITATIONS:
no limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-12

comments: SUMMARY:
This paper studies the unsupervised learning for high-dimensional halfspace under affine transformation. The goal is to estimate the normal vector of the halfspace such that the learned distribution is close to the underlying truth. The paper provides an algorithm using so-called ''contrastive moments'' with sample and time complexity guarantee.

STRENGTHS:
- The algorithm provided achieves polynomial dependence in $d, 1/\epsilon, 1/\delta$ for both time and sample complexity.
- Existing literature relies on supervised learning, or Gaussianity of the distribution, or NGCA and ICA. While this paper only requires log-concavity.
- Techniques introduced, e.g. contrastive moments, might be of independent interest.

WEAKNESSES:
- The product and isotropic distribution with affine transformation assumption seems to be restrictive, can the analysis or algorithm go beyond that?
- The presentation of the problem is hard to follow, might need more explicit mathematical definitions. For example, where is $\tilde{P}$ in **Problem** defined.

QUESTIONS:
- Are coordinates of $P(Q)$ independent with each other before the affine transformation?
- Can author(s) provide an real data application example for the problem setup in Definition 1, and specify the the roles of $q,Q,d, P,u$?

LIMITATIONS:
No limitation is discussed.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-13