Title: Forster Decomposition and Learning Halfspaces with Noise
Authors: Ilias Diakonikolas
Daniel Kane
Christos Tzamos
Venue: NeurIPS 2021
Keywords: learning theory
Forster transform
halfspaces
Massart noise

Abstract: A Forster transform is an operation that turns a multivariate distribution into one with good anti-concentration properties. While a Forster transform does not always exist, we show that any distribution can be efficiently decomposed as a disjoint mixture of few distributions for which a Forster transform exists and can be computed efficiently. As the main application of this result, we obtain the first polynomial-time algorithm for distribution-independent PAC learning of halfspaces in the Massart noise model with strongly polynomial sample complexity, i.e., independent of the bit complexity of the examples. Previous algorithms for this learning problem incurred sample complexity scaling polynomially with the bit complexity, even though such a dependence is not information-theoretically necessary.


TL;DR: First efficient learning algorithm for Massart halfspaces with sample complexity independent of the bit complexity of the examples.

Review discussion:
**Originality**
The results in this paper are certainly novel and is an important improvement over prior works. The techniques are nice and I am not aware of other works which make use of Forster transformations. The techniques may also warrant further study and see where it may also be applicable.
**Quality**
The submission is technically sound as far as I can tell although I did not carefully verify any of the arguments. All the claims have been justified either in the main body or in the appendix.
**Clarity**
The paper is written quite well. There are quite a few ideas in the paper including the various aspects of the Forster transform and its application to PAC learning halfspaces with Massart noise.
However, the authors did a pretty decent job explaining their approach.
**Significance**
The results are quite nice and make a significant contribution to the literature on learning under Massart noise.
Previously, a strongly polynomial sample complexity was known but only information theoretically and this paper makes it (weakly) algorithmic. They also point out that obtaining a (strongly) polynomial time algorithm for the problem would also result in a strongly polynomial time algorithm for linear programming.
The results in the paper may be of interest to researchers working in learning theory and, in particular, researchers that are working on robust PAC learning algorithms. The techniques are also very nice and may also interest other researchers working in the area.
**Other comments**
I would appreciate if the authors compared the actual sample complexity.
In particular, what are the lower bounds, what they achieve, and perhaps compare this to previous works.
No
We thank the reviewer for their positive feedback. We addressed the significance of our results in the joint response to the reviewers.
Here we answer a specific question by the reviewer.
The information-theoretic sample complexity of learning halfspaces in d dimensions
within error eps under eta-Massart noise is Theta (min { d / (1-2 eta) eps , d/eps^2}),
as shown in the work of Massart and Nedelec [MN06].
Yet the only known computationally efficient algorithms by [DGT19] and, subsequently [CKMY20],
achieve error guarantee of eta+eps with sample complexity that scales (polynomially)
on the bit-complexity of the points. The degree of the polynomial dependence in all parameters
is suboptimal in these algorithmic upper bounds.
While it is not clear whether one can achieve the information-theoretically optimal sample complexity
in polynomial time, even for the relaxed error guarantees, our work shows that it is possible
to efficiently learn with poly(d/((1-2eta)eps)) samples --- thus eliminating the dependence
on the bit complexity. In that sense, our improvement is more qualitative rather than quantitative.
Thanks for the response. I will be leaving my evaluation unchanged.
If I understand correctly, the new ideas in this paper are the following.
First is identifying the connection between radial isotropic position (more specifically the structural result in [HKLM20]) and preprocessing algorithms like Dunagan-Vempala for learning halfspaces without margin.
The second is the following observation about [HKLM20]. For context, the [HKLM20] paper was non-constructive in the sense that for the "hard" direction (failure of a dataset to be placed in radial isotropic position implies existence of a hard subspace), they invoked Barthe's theorem, together with separating hyperplane theorem, to conclude that the LP in eq. 3 of the submission is feasible. From there, [HKLM20] showed how to identify a heavy subspace from a feasible point using essentially the same argument as in lines 281-288 of the submission. The main innovation, in Proposition 2.3, is to realize that this latter part of the argument in [HKLM20] yields an efficient algorithm for identifying heavy subspaces/certifying none exist.
Once you have this, you can find a heavy subspace W with no heavy subspaces of its own (which the authors do in Lines 298-300). At this point though, couldn't you also just run the algorithm of Hardt-Moitra to put the points in W in radial isotropic position, because you're guaranteed by [HKLM20] that such a transformation exists? I'm not sure if Proposition 2.2 or Lines 300-305 are needed, but perhaps I'm missing something.
Overall, I think the application of Theorem 1.4 to learning halfspaces under Massart noise is a nice observation.
In terms of Theorem 1.4 however, I would regard the main contribution of that result as primarily synthesizing existing ideas (in a clever way). This is not necessarily a weakness, but I think the submission could be strengthened if the authors spelled out more clearly where the ideas in the proof of their main Theorem 1.4 come from and, e.g., clarified the connection to Hardt-Moitra.
Minor typos:
Line 210: should be Bv^T \cdot \Sigma v
Lines 211-215 "if c(x) is too small, the expectation of c(X)^2XX^T will fail to add up to Sigma" -> this was a bit hard to parse because Sigma is defined in terms of c(x)
Line 283: x^{(i_d)} should be x^{(i_k)}
Line 288: contradiction -contradiction-
Display above Line 288: \sum_{x_i\in \mathcal{B}} instead of \sum_{x_i\in B}
No
We would like to thank the reviewer for their feedback. We address a specific question by the reviewer below.
We start by pointing out that the main contribution of our work is conceptual. The high-level message of our work is
that Forster decompositions provide the "correct" pre-processing procedure to eliminate the bit complexity
from the sample size for the problem of efficiently learning halfspaces with Massart noise. In more detail:
- We relax the notion of the Forster transform to a Forster decomposition, so that it applies to any point-set,
and we provide an efficient algorithm to compute such a decomposition; and
- We provide a novel application of this technique of Forster decompositions in a well-studied learning setting,
giving the first computationally efficient algorithm with strongly-polynomial sample complexity
for learning halfspaces in the presence of Massart noise.
The prior results of [AKS20] and [HM13] do not provide the desirable guarantees when a Forster transform does not exist,
which is crucial for our learning application.
In particular, our Proposition 2.3 is new and not covered by these prior works.
On the other hand, Proposition 2.2 is not a major contribution of our work, and its (short) proof is given mainly
for the sake of the presentation.
In particular, the algorithm of [AKS20] or [HM13] could have been used
in place of our SDP formulation in Proposition 2.2 for computing Forster transforms when no heavy subspace exists.
(We will clarify this point in the final version of the paper.)
However, the application of these results in our setting would require additional work to establish
that the required conditions for computing such a transform are satisfied by point-sets with bounded bit complexity.
As a side note, we note that the analysis of [HM13] actually has a bug, pointed out and corrected in [AKS20].
To the best of our understanding, the analysis in [AKS20] is correct and their main theorem could have been used
in place of our Proposition 2.2, modulo the aforementioned technicalities. For the sake of readability, we wanted
to provide a self-contained version of the algorithm itself.
Moreover, we believe that the direct SDP formulation given in our
Proposition 2.2, which is different from the approach taken in the aforementioned prior works, is more succinct
and intuitive, and facilitates the presentation of our results.
Thanks for the clarification regarding [HM13]/[AKS20]! And the conceptual contribution of identifying radial isotropic position as the right pre-processing tool for this problem is indeed quite nice. After reading the other reviews, I think I agree with the sentiment that the result is somewhat niche, but I still think this paper would be a solid addition to the program and thus will leave my score unchanged.
**Pros:**
1. The result is technically strong, probably among the top theory papers submitted in this axis. The Forster decomposition it computes seems like a natural object that might be used elsewhere. The idea that on can always rescale for homogenous halfspaces is simple and nice.
2. It gives an improvement in the sample complexity for learning halfspaces with Massart noise, getting rid of the bit-size dependence.
**Cons:**
1. The result is rather technical and narrowly focused. I am not sure how many people who weren't working on this problem were aware of the dependence of the sample complexity on bit size in the previous work.  This will be exciting to the experts in the area, but I have a hard time imaging a broader audience. It is not the kind of theory result that I would want the broader ML community to hear about, which in my opinion makes it a better fit for a more theory-focused venue like STOC/FOCS/COLT.
Having said that, I know NeurIPS aims to have a broad tent, so perhaps the PC would like to make room for a few more narrowly focused papers.
2. The quality of writing leaves a lot to be desired. The high level explanations are ok, but the choice of notation and technical writing seems rather hurried. See some concrete suggestions below:
- Using $\sigma$ for number of standard deviations is a strange choice with potential for confusion.
- The definition of radial position reads as though we want $E[xx^T] = I/d$ for every point $x$. You should separate  out the conditions that are needed in expectation from those that must hold pointwise.
- In the statement of Theorem 1.4, do you need any non-trivial lower bound on $dim(V)$? Or would $dim(V) =1$ be fine? I suspect it is the latter, but you should make this explicit.
- The paper points  to Theorem 3.4 for a detailed statement of 1.4.
But Theorem 3.4 talks about the learning application.
- I would have liked to see a Theorem statement where you unwind the recursive use of 1.4 and state the decomposition of the original point set into subspaces.
No
We thank the reviewer for their positive feedback regarding the technical strength of our results.
We addressed the fit to NeurIPS in the joint response to the reviewers.
In summary, learning halfspaces with Massart noise is a classical problem in learning theory.
Learning theory is an important subarea of NeurIPS and well within the scope of the conference.
Interestingly, the prior work on this specific problem (that we improve upon) was awarded the best paper in NeurIPS 2019,
and there has been substantial work in the learning theory community since. Our work *qualitatively* improves
on this NeurIPS 2019 paper, achieving strongly-polynomial sample complexity for the problem.
Underlying the algorithms in this paper is a line of work wherein many points are guaranteed to be far from the separating hyperplane. Without knowledge of the true separator, this can still be ensured by alinear transform which ensures some bound on the expectation of dot products, while their absolute values are still bounded.
While previous approaches used a combination of linear transforms and removal of bad points, the approach here is slightly different: joining a linear transform and a scaling step, resulting in the points being approximate in a radial isotropic formation. This transformation can be computed, and applied iteratively to the point set in order to ensure that many points are far from the final separating hyperplane.
A central step in the proof gives an LP, which can be solved efficiently using the ellipsoid method. This gives a dependence on the bit precision in runtime (as observed to sample complexity), but is likely unavoidable.
I found this paper to be an enjoyable and interesting read, as well as mathematically challenging. My reservations are based on the application, that is eliminating sample complexity dependence on bit precision within a very specific noise model. This seems to be a lot of work to achieve a relatively minor improvement. Are there other applications for these techniques?
No
We thank the reviewer for their feedback and consideration of our work.
We are encouraged by the fact that the reviewer found our work enjoyable
and mathematically challenging.
In the joint response to the reviewers, we address the significance of our contributions in detail.
In short, while the improvement to the sample complexity might at first sight seem minor,
going from weakly polynomial to strongly polynomial is both conceptually and qualitatively important.
Beyond this application, the technique of Forster Decompositions can be useful in other contexts as well.
Comment: We thank the reviewers for their time and effort in providing feedback. In our joint response, we would like to address some common questions on the significance of our contributions and their fit to NeurIPS.
The contribution of our work is two-fold:
1) We develop a novel technique of Forster decompositions that relaxes the notion of the Forster transform,
so that it applies to any point-set; and we provide an efficient algorithm to compute such a decomposition.
2) We provide a novel application of this technique to a well-studied learning setting, giving the first computationally
efficient algorithm with strongly-polynomial sample complexity for learning halfspaces in the presence of Massart noise.
We would like to point out that learning halfspaces with Massart noise is a classical problem in learning theory.
Learning theory is an important subarea of NeurIPS and well within the scope of the conference.
Interestingly, the prior work on this specific problem (that we improve upon) was awarded the best paper in NeurIPS 2019,
and there has been substantial work in the learning theory community since. Our work *qualitatively* improves
on this NeurIPS 2019 paper, achieving strongly-polynomial sample complexity for the problem.
This qualitative improvement is enabled by connecting the problem to the literature on radial isotropic transformations.
Before our work, a large number of results were based on filtering points given in the input to ensure that no "outliers" exist,
and as a result paid an overhead that was reflected in the weakly polynomial dependence on the sample complexity.
Conceptually, our approach implies that one should not throw away points (at least for the problem of learning halfspaces);
and that with appropriate *reweighting* they can all be useful for learning. Beyond this application, we also note that our technique
of Forster Decompositions and our algorithmic results can be useful in a variety of other contexts.
Specifically, they imply
improvements to the problem of subspace recovery and can be useful as a preconditioning step in other learning tasks
where one can rotate points and scale them without changing the task.

Meta-review:
role: PC
data: comment: This paper proposes Forster decomposition, a new analysis tool for learning linear classifiers and applies it to derive an algorithm that learns halfspaces with Massart noise without the dependence on bit complexity (which was necessary for previous algorithms). The contributions is very strong technically and addresses one of the classical problems in ML. While it has downsides, such as being technically rather complex and likely to be fully appreciated by a handful of experts, I recommend acceptance.

Decision: Accept (Spotlight)