Forster Decomposition and Learning Halfspaces with Noise

Abstract: A Forster transform is an operation that turns a multivariate distribution into one with good anti-concentration properties. While a Forster transform does not always exist, we show that any distribution can be efficiently decomposed as a disjoint mixture of few distributions for which a Forster transform exists and can be computed efficiently. As the main application of this result, we obtain the first polynomial-time algorithm for distribution-independent PAC learning of halfspaces in the Massart noise model with strongly polynomial sample complexity, i.e., independent of the bit complexity of the examples. Previous algorithms for this learning problem incurred sample complexity scaling polynomially with the bit complexity, even though such a dependence is not information-theoretically necessary.


Reviews and author responses:

 Thanks for the response. I will be leaving my evaluation unchanged.

 Thanks for the clarification regarding [HM13]/[AKS20]! And the conceptual contribution of identifying radial isotropic position as the right pre-processing tool for this problem is indeed quite nice. After reading the other reviews, I think I agree with the sentiment that the result is somewhat niche, but I still think this paper would be a solid addition to the program and thus will leave my score unchanged.

 We thank the reviewer for their positive feedback. We addressed the significance of our results in the joint response to the reviewers.
Here we answer a specific question by the reviewer.

The information-theoretic sample complexity of learning halfspaces in d dimensions 
within error eps under eta-Massart noise is Theta (min { d / (1-2 eta) eps , d/eps^2}), 
as shown in the work of Massart and Nedelec [MN06]. 
Yet the only known computationally efficient algorithms by [DGT19] and, subsequently [CKMY20], 
achieve error guarantee of eta+eps with sample complexity that scales (polynomially) 
on the bit-complexity of the points. The degree of the polynomial dependence in all parameters 
is suboptimal in these algorithmic upper bounds. 

While it is not clear whether one can achieve the information-theoretically optimal sample complexity 
in polynomial time, even for the relaxed error guarantees, our work shows that it is possible 
to efficiently learn with poly(d/((1-2eta)eps)) samples --- thus eliminating the dependence 
on the bit complexity. In that sense, our improvement is more qualitative rather than quantitative.


 We would like to thank the reviewer for their feedback. We address a specific question by the reviewer below.

We start by pointing out that the main contribution of our work is conceptual. The high-level message of our work is
that Forster decompositions provide the "correct" pre-processing procedure to eliminate the bit complexity
from the sample size for the problem of efficiently learning halfspaces with Massart noise. In more detail:
- We relax the notion of the Forster transform to a Forster decomposition, so that it applies to any point-set,
and we provide an efficient algorithm to compute such a decomposition; and
- We provide a novel application of this technique of Forster decompositions in a well-studied learning setting,
giving the first computationally efficient algorithm with strongly-polynomial sample complexity
for learning halfspaces in the presence of Massart noise.

The prior results of [AKS20] and [HM13] do not provide the desirable guarantees when a Forster transform does not exist,
which is crucial for our learning application. In particular, our Proposition 2.3 is new and not covered by these prior works.

On the other hand, Proposition 2.2 is not a major contribution of our work, and its (short) proof is given mainly
for the sake of the presentation. In particular, the algorithm of [AKS20] or [HM13] could have been used
in place of our SDP formulation in Proposition 2.2 for computing Forster transforms when no heavy subspace exists.
(We will clarify this point in the final version of the paper.)
However, the application of these results in our setting would require additional work to establish
that the required conditions for computing such a transform are satisfied by point-sets with bounded bit complexity.
As a side note, we note that the analysis of [HM13] actually has a bug, pointed out and corrected in [AKS20].
To the best of our understanding, the analysis in [AKS20] is correct and their main theorem could have been used
in place of our Proposition 2.2, modulo the aforementioned technicalities. For the sake of readability, we wanted
to provide a self-contained version of the algorithm itself. Moreover, we believe that the direct SDP formulation given in our
Proposition 2.2, which is different from the approach taken in the aforementioned prior works, is more succinct
and intuitive, and facilitates the presentation of our results.


 We thank the reviewer for their positive feedback regarding the technical strength of our results.
We addressed the fit to NeurIPS in the joint response to the reviewers.

In summary, learning halfspaces with Massart noise is a classical problem in learning theory.
Learning theory is an important subarea of NeurIPS and well within the scope of the conference.
Interestingly, the prior work on this specific problem (that we improve upon) was awarded the best paper in NeurIPS 2019,
and there has been substantial work in the learning theory community since. Our work *qualitatively* improves
on this NeurIPS 2019 paper, achieving strongly-polynomial sample complexity for the problem.

 We thank the reviewer for their feedback and consideration of our work.
We are encouraged by the fact that the reviewer found our work enjoyable 
and mathematically challenging.

In the joint response to the reviewers, we address the significance of our contributions in detail. 
In short, while the improvement to the sample complexity might at first sight seem minor, 
going from weakly polynomial to strongly polynomial is both conceptually and qualitatively important. 
Beyond this application, the technique of Forster Decompositions can be useful in other contexts as well. 


 Comment: We thank the reviewers for their time and effort in providing feedback. In our joint response, we would like to address some common questions on the significance of our contributions and their fit to NeurIPS.

The contribution of our work is two-fold:
1) We develop a novel technique of Forster decompositions that relaxes the notion of the Forster transform,
so that it applies to any point-set; and we provide an efficient algorithm to compute such a decomposition.
2) We provide a novel application of this technique to a well-studied learning setting, giving the first computationally
efficient algorithm with strongly-polynomial sample complexity for learning halfspaces in the presence of Massart noise.

We would like to point out that learning halfspaces with Massart noise is a classical problem in learning theory.
Learning theory is an important subarea of NeurIPS and well within the scope of the conference.
Interestingly, the prior work on this specific problem (that we improve upon) was awarded the best paper in NeurIPS 2019,
and there has been substantial work in the learning theory community since. Our work *qualitatively* improves
on this NeurIPS 2019 paper, achieving strongly-polynomial sample complexity for the problem.

This qualitative improvement is enabled by connecting the problem to the literature on radial isotropic transformations.
Before our work, a large number of results were based on filtering points given in the input to ensure that no "outliers" exist,
and as a result paid an overhead that was reflected in the weakly polynomial dependence on the sample complexity.
Conceptually, our approach implies that one should not throw away points (at least for the problem of learning halfspaces);
and that with appropriate *reweighting* they can all be useful for learning. Beyond this application, we also note that our technique
of Forster Decompositions and our algorithmic results can be useful in a variety of other contexts. Specifically, they imply
improvements to the problem of subspace recovery and can be useful as a preconditioning step in other learning tasks
where one can rotate points and scale them without changing the task.

This paper shows that any distribution can be decomposed into a number of disjoint distributions, each admitting a Forster transform, which has anti-concentration properties. An application of this decomposition is to learning halfspaces in the Massart noice model.  Underlying the algorithms in this paper is a line of work wherein many points are guaranteed to be far from the separating hyperplane. Without knowledge of the true separator, this can still be ensured by alinear transform which ensures some bound on the expectation of dot products, while their absolute values are still bounded. While previous approaches used a combination of linear transforms and removal of bad points, the approach here is slightly different: joining a linear transform and a scaling step, resulting in the points being approximate in a radial isotropic formation. This transformation can be computed, and applied iteratively to the point set in order to ensure that many points are far from the final separating hyperplane.

A central step in the proof gives an LP, which can be solved efficiently using the ellipsoid method. This gives a dependence on the bit precision in runtime (as observed to sample complexity), but is likely unavoidable.

I found this paper to be an enjoyable and interesting read, as well as mathematically challenging. My reservations are based on the application, that is eliminating sample complexity dependence on bit precision within a very specific noise model. This seems to be a lot of work to achieve a relatively minor improvement. Are there other applications for these techniques? Yes

This paper presents an algorithm to compute a certain decomposition of a point set known as a Forster decomposition. Roughly speaking this guarantees 
1. mean 0 and  close identity covariance
2. all points with norm close to one.
Each of these conditions is easy to achieve (via PCA and rescaling respectively),  the challenge is getting them to hold simultaneously.
The kind of mapping they seeks is an invertiable linear transformation followed by rescaling. 

While such a mapping may not always exist, it was known that the only "obstructions" are dense subspaces (in the extreme think of the same point repeated many times). The main algorithm in this paper allows one to find these obstructions efficiently. One can then iteratively  remove these points and keep going. This gives a decomposition into O(d) many subspaces and a good mapping for the points within each subspace.

The main application of this result is a learning algorithm for halfspaces in the Massart noise model whose sample complexity is independent of the bit-length of the inputs. The running time still depends polynomially on the bit lengths (inevitable by a reduction from LP). They key idea is that the mapping they use preserves the existance of a good homogenous halfspace classifier.
  **Pros:**
1. The result is technically strong, probably among the top theory papers submitted in this axis. The Forster decomposition it computes seems like a natural object that might be used elsewhere. The idea that on can always rescale for homogenous halfspaces is simple and nice. 

2. It gives an improvement in the sample complexity for learning halfspaces with Massart noise, getting rid of the bit-size dependence. 

**Cons:**
1. The result is rather technical and narrowly focused. I am not sure how many people who weren't working on this problem were aware of the dependence of the sample complexity on bit size in the previous work.  This will be exciting to the experts in the area, but I have a hard time imaging a broader audience. It is not the kind of theory result that I would want the broader ML community to hear about, which in my opinion makes it a better fit for a more theory-focused venue like STOC/FOCS/COLT. Having said that, I know NeurIPS aims to have a broad tent, so perhaps the PC would like to make room for a few more narrowly focused papers. 

2. The quality of writing leaves a lot to be desired. The high level explanations are ok, but the choice of notation and technical writing seems rather hurried. See some concrete suggestions below:

- Using $\sigma$ for number of standard deviations is a strange choice with potential for confusion.
- The definition of radial position reads as though we want $E[xx^T] = I/d$ for every point $x$. You should separate  out the conditions that are needed in expectation from those that must hold pointwise.
- In the statement of Theorem 1.4, do you need any non-trivial lower bound on $dim(V)$? Or would $dim(V) =1$ be fine? I suspect it is the latter, but you should make this explicit. 
- The paper points  to Theorem 3.4 for a detailed statement of 1.4. But Theorem 3.4 talks about the learning application. 
- I would have liked to see a Theorem statement where you unwind the recursive use of 1.4 and state the decomposition of the original point set into subspaces.



 Given the very abstract nature of the problem they are addressing, I don't think there is much to say. 

This paper studies the problem of learning halfspaces under Massart noise in the absence of distributional assumptions on the x's. Previous works have given algorithms for this problem which achieve error eta + eps in time and sample complexity polynomial in d, 1/eps, and the bit complexity b of the x's. Removing the dependence on bit complexity in the runtime for the closely related problem of linear programming is a major open question, but more modestly one could hope to remove this dependence in the sample complexity, and this is the main result of this paper, which it achieves using a modification of the improper learner from [DGT19].

The main bottleneck that leads to the b dependence in previous algorithms is a preprocessing step due to Dunagan-Vempala that essentially reduces to the large margin case by removing some outliers so that no point among the remaining points S is far from S in Mahalanobis. The present work circumvents this as follows. For the preprocessing step, instead of outlier removal, one could also just try placing the dataset in approximate radial isotropic position. This is a technique that has been useful in a variety of other applications like subspace recovery, the Paulsen problem, communication complexity, etc. 

Unfortunately, not every dataset can be placed in radial isotropic position, for instance if some m-dimensional subspace is "heavy" in the sense that it contains more than an m/d fraction of the points. As shown in [HKLM20], this is the only obstruction, and the main contribution of the present work is to make this fact algorithmic, i.e. to give an algorithm for finding a heavy subspace such that the part of the dataset in that subspace can be placed in radial isotropic position.

Then to get the application to learning under Massart noise, whenever one needs to run the outlier removal step, just use the above algorithmic result to identify a heavy subspace, place those points in radial isotropic position, and run one step of the algorithm for learning large-margin halfspaces under Massart noise on those points to make progress.  If I understand correctly, the new ideas in this paper are the following. First is identifying the connection between radial isotropic position (more specifically the structural result in [HKLM20]) and preprocessing algorithms like Dunagan-Vempala for learning halfspaces without margin.

The second is the following observation about [HKLM20]. For context, the [HKLM20] paper was non-constructive in the sense that for the "hard" direction (failure of a dataset to be placed in radial isotropic position implies existence of a hard subspace), they invoked Barthe's theorem, together with separating hyperplane theorem, to conclude that the LP in eq. 3 of the submission is feasible. From there, [HKLM20] showed how to identify a heavy subspace from a feasible point using essentially the same argument as in lines 281-288 of the submission. The main innovation, in Proposition 2.3, is to realize that this latter part of the argument in [HKLM20] yields an efficient algorithm for identifying heavy subspaces/certifying none exist. 

Once you have this, you can find a heavy subspace W with no heavy subspaces of its own (which the authors do in Lines 298-300). At this point though, couldn't you also just run the algorithm of Hardt-Moitra to put the points in W in radial isotropic position, because you're guaranteed by [HKLM20] that such a transformation exists? I'm not sure if Proposition 2.2 or Lines 300-305 are needed, but perhaps I'm missing something.

Overall, I think the application of Theorem 1.4 to learning halfspaces under Massart noise is a nice observation. In terms of Theorem 1.4 however, I would regard the main contribution of that result as primarily synthesizing existing ideas (in a clever way). This is not necessarily a weakness, but I think the submission could be strengthened if the authors spelled out more clearly where the ideas in the proof of their main Theorem 1.4 come from and, e.g., clarified the connection to Hardt-Moitra.

Minor typos:
Line 210: should be Bv^T \cdot \Sigma v
Lines 211-215 "if c(x) is too small, the expectation of c(X)^2XX^T will fail to add up to Sigma" -> this was a bit hard to parse because Sigma is defined in terms of c(x)
Line 283: x^{(i_d)} should be x^{(i_k)}
Line 288: contradiction -contradiction-
Display above Line 288: \sum_{x_i\in \mathcal{B}} instead of \sum_{x_i\in B} Yes, this work does not have any potential negative societal impact.

This paper studies the problem of finding a Forster transformation of a given distribution. The goal of a Foster transform is to transform an arbitrary distribution into a distribution that has good anti-concentration properties: in particular, one would like the transformed distribution to have its second moment matrix equal to a scaled identity matrix and the points from the distribution should also have Euclidean norm 1. For this problem, they give an algorithm that runs in weakly polynomial time. The algorithm itself uses the ellipsoid algorithm to solve SDPs and LPs.

As an application of their algorithm for finding Forster transforms, they consider the problem of PAC learning halfspaces under Massart noise. They show that there is an algorithm whose sample complexity is strongly polynomial and running time is weakly polynomial. Previously, the work of Diakonikolas et al. showed that there is an algorithm whose sample complexity and running are both weakly polynomial whereas, information-theoretically, it is known that the sample complexity can be strongly polynomial. So the present paper closes this gap up to polynomial factors.  **Originality**
The results in this paper are certainly novel and is an important improvement over prior works. The techniques are nice and I am not aware of other works which make use of Forster transformations. The techniques may also warrant further study and see where it may also be applicable.

**Quality**
The submission is technically sound as far as I can tell although I did not carefully verify any of the arguments. All the claims have been justified either in the main body or in the appendix.

**Clarity**
The paper is written quite well. There are quite a few ideas in the paper including the various aspects of the Forster transform and its application to PAC learning halfspaces with Massart noise. However, the authors did a pretty decent job explaining their approach.

**Significance**
The results are quite nice and make a significant contribution to the literature on learning under Massart noise. Previously, a strongly polynomial sample complexity was known but only information theoretically and this paper makes it (weakly) algorithmic. They also point out that obtaining a (strongly) polynomial time algorithm for the problem would also result in a strongly polynomial time algorithm for linear programming.

The results in the paper may be of interest to researchers working in learning theory and, in particular, researchers that are working on robust PAC learning algorithms. The techniques are also very nice and may also interest other researchers working in the area.

**Other comments**
I would appreciate if the authors compared the actual sample complexity. In particular, what are the lower bounds, what they achieve, and perhaps compare this to previous works. Yes

Meta-review: This paper proposes Forster decomposition, a new analysis tool for learning linear classifiers and applies it to derive an algorithm that learns halfspaces with Massart noise without the dependence on bit complexity (which was necessary for previous algorithms). The contributions is very strong technically and addresses one of the classical problems in ML. While it has downsides, such as being technically rather complex and likely to be fully appreciated by a handful of experts, I recommend acceptance.

Decision: accept