ReLU Regression with Massart Noise

1 Introduction . Learning in the presence of outliers is a key challenge in machine learning , with several data analysis applications , including in ML security ( 4 ; 7 ; 46 ; 14 ) and in exploratory data analysis of real datasets with natural outliers , e.g. , in biology ( 43 ; 41 ; 36 ) . The goal is to design computationally efficient learners that can tolerate a constant fraction of outliers , independent of the dimensionality of the data . Early work in robust statistics ( 28 ; 30 ) gave sample-efficient robust estimators for various basic tasks , alas with exponential runtime . A recent line of work in computer science , starting with ( 13 ; 35 ) , developed the first computationally efficient robust learning algorithms for various high-dimensional tasks . Since these early works , there has been significant progress in algorithmic robust high-dimensional statistics by several communities , see ( 16 ) for a recent survey . In this work , we study the problem of learning Rectified Linear Units ( ReLUs ) in the presence of label noise . The ReLU function ReLUw : Rd ! R , parameterized by a vector w 2 Rd , is defined as ReLUw ( x ) : = ReLU ( w · x ) = max { 0 , w · x } . ReLU regression – the task of fitting ReLUs to a set of labeled examples – is a fundamental task and an important primitive in the theory of deep learning . In recent years , ReLU regression has been extensively studied in theoretical machine learning both from the perspective of designing efficient algorithms and from the perspective of computational hardness , see , e.g. , ( 26 ; 45 ; 37 ; 48 ; 27 ; 49 ; 11 ; 15 ; 25 ; 18 ) . The computational difficulty of this problem crucially depends on the assumptions about the input data . In the realizable case , i.e. , when the labels are consistent with the target function , the problem is efficiently solvable , see , e.g. , ( 45 ) . On the other hand , in the presence of even a small constant fraction of adversarially labeled data , computational hardness results are known even for approximate recovery ( 29 ; 37 ) and under well-behaved distributions ( 27 ; 15 ; 25 ; 18 ) . See Section 1.3 for a detailed summary of related work . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . A challenging corruption model is the adversarial label noise model , in which an adversary is allowed to corrupt an arbitrary ⌘ < 1/2 fraction of the labels . The aforementioned hardness results rule out the existence of efficient algorithms for learning ReLUs with optimal error guarantees in this model , even when the underlying distribution on examples is Gaussian . Moreover , when no assumptions are made on the underlying data distribution , no fully polynomial time algorithm with non-trivial guarantee is possible . In fact , for the distribution-independent setting , even for the simpler case of learning linear functions , there are strong computational hardness results for any constant ⌘ ( 33 ; 29 ) . These negative results motivate the following natural question : Are there realistic label noise models in which efficient learning is possible without strong distributional assumptions ? Here we focus on ReLU regression in the presence of Massart ( or bounded ) noise ( 38 ) , and provide an efficient learning algorithm with minimal distributional assumptions . In the process , we also provide an efficient noise-tolerant algorithm for the simpler case of linear regression . The Massart model ( 38 ) is a classical semi-random noise model originally defined in the context of binary classification . In this model , an adversary has control over a random ⌘ < 1/2 fraction of the labels . Recent work ( 12 ) gave the first efficient learning algorithm for linear separators with non-trivial error gurrantees in the Massart model without distributional assumptions . In this work , we ask to what extent such algorithmic results are possible for learning real-valued functions . To state our contributions , we formally define the following natural generalization of the model for real-valued functions . Definition 1.1 ( Learning Real-valued Functions with Massart Noise ) . Let F be a concept class of real-valued functions over Rd and f : Rd ! R be an unknown function in F . For a given parameter ⌘ < 1/2 , the algorithm specifies m 2 Z+ and obtains m samples ( xi , yi ) mi=1 , such that : ( a ) every xi is drawn i.i.d . from a fixed distribution Dx , and ( b ) each yi is equal to f ( xi ) with probability 1 ⌘ and takes an arbitrary value with probability ⌘ , chosen by an adversary after observing the samples drawn and the values that can be corrupted . In the context of binary classification , the above model has been extensively studied in the theoretical ML community for the class of linear separators ( 2 ; 3 ; 51 ; 50 ; 20 ; 12 ; 9 ) . Even though the noise model might appear innocuous at first sight , the ability of the Massart adversary to choose whether to perturb a given label and , if so , with what probability ( which is unknown to the learner ) , makes the design of efficient algorithms in this model challenging . Specifically , for distribution-independent PAC learning of linear separators , even approximate learning in this model is computationally hard ( 17 ) . Extending this model to real-valued functions , we study regression under Massart noise for the realizable setting ; that is , when the uncorrupted data exhibit clean functional dependencies , i.e. , yi = ReLU ( w⇤ · xi ) . The realizable setting is both of theoretical and practical interest . Prior work ( 45 ; 23 ; 31 ; 49 ) developed algorithms for learning ReLUs in this setting ( without Massart noise ) , providing theoretical insights on the success of deep learning architectures . On the practical side , there are many applications in which we observe clean functional dependencies on the uncorrupted data . For instance , clean measurements are prevalent in many signal processing applications , including medical imaging , and are at the heart of the widely popular field of compressive sensing ( 8 ) . 1.1 Main Results . To build up to the more challenging case of ReLUs , we start with the simpler case of linear functions , which is in and of itself one of the most well-studied statistical tasks , with numerous applications in machine learning ( 44 ) , as well as in other disciplines , including economics ( 21 ) and biology ( 39 ) . In our Massart setting , the goal is to identify a linear relation y = w⇤ · x that the clean samples ( x , y ) ( inliers ) satisfy . We show that , under the minimal ( necessary ) assumption that the distribution is not fully concentrated on any subspace , the problem is efficiently identifiable . Theorem 1.2 ( Exact Recovery of Linear Functions ) . Let Dx be a distribution on Rd that has zero measure on any linear subspace and let ⌘ < 1/2 be the upper bound on the Massart noise rate . Denote w⇤ the vector representing the true linear function . There is an algorithm that draws Õ ( d 3 ( 1 2⌘ ) 2 ) samples , runs in poly ( d , b , ( 1 2⌘ ) 1 ) time , where b is an upper bound on the bit complexity of the samples and parameters , and outputs w⇤ with probability at least 9/10 . We provide a more general algorithmic result that relaxes the density assumption on Dx in Theorem C.3 of Appendix C , so that the only assumption needed is that the support of Dx spans Rd . It is important to note that if the distribution was concentrated entirely on a linear subspace , it would be information-theoretically impossible to determine the orthogonal component of w⇤ on that subspace . That is , our anti-concentration assumption on the distribution is necessary for identifiability to be possible . Even when this assumption is violated and the problem is non-identifiable , we provide a ( weaker ) PAC learning guarantee for the linear case in Theorem D.1 of Appendix D. Our main algorithmic result is for the problem of ReLU regression , where the inliers satisfy y = ReLU ( w⇤ · x ) and an ⌘ < 1/2 fraction of the labels are corrupted by Massart noise . Even in this more challenging case , we show it is possible to efficiently identify the true parameters w⇤ , as long as every homogeneous halfspace contains a non-negligible fraction of the sample points . Theorem 1.3 ( Exact Recovery of ReLUs ) . Let Dx be a distribution on Rd that has zero measure on any linear subspace and such that Prx⇠Dx [ w · x 0 ] for any w 2 Rd . Let ⌘ < 1/2 be the upper bound on the Massart noise rate . Denote w⇤ the parameter vector of the target ReLU . There is an algorithm that draws Õ ( d 3 2 ( 1 2⌘ ) 2 ) samples , runs in time poly ( d , b , 1 , ( 1 2⌘ ) 1 ) , and outputs w⇤ with probability at least 9/10 . We similarly provide a more general result relaxing the density assumption on Dx in Theorem C.4 of Appendix C. We note that the assumption on the mass of halfspaces in Theorem 1.3 is necessary for identifiability . Indeed , if there was a halfspace , parameterized by w 2 Rd , such that Prx⇠Dx [ w ·x 0 ] = 0 , it would be impossible to distinguish between the functions ReLU ( w · x ) and ReLU ( 2w · x ) ( even without noise ) , as all points would have 0 labels . It remains an interesting open problem whether similar PAC learning guarantees can be obtained for the case of ReLU regression . We suspect that this problem is computationally hard in full generality . 1.2 Technical Overview . As explained in the introduction , the focus of our work is on the problem of robust regression in the presence of outliers . When ⌘ < 12 fraction of the data is arbitrarily corrupted in the realizable setting , the goal is to compute the function that fits as many as points ( inliers ) as possible . Given a sufficient number of samples from a full-dimensional distribution , this function is unique for the class of ReLUs and matches the true function with high probability . However , even in the simpler case of linear functions , the corresponding computational problem of ` 0-minimization is computationally hard without distributional assumptions , as it is an instance of robust subspace recovery ( 29 ) . Our positive results are driven by relaxing the assumption that an arbitrary ⌘ fraction of the points is corrupted . Instead , as defined in Definition 1.1 , we consider a more restricted adversary that is presented with a uniformly random ⌘ fraction of the points , which can be corrupted arbitrarily at will . ` 0 to ` 1 minimization Given this milder corruption model , we propose novel algorithms for efficient exact recovery of the underlying function . We obtain our algorithms by replacing the ` 0-minimization with ` 1-minimization , which can be shown to converge to the true function in the limit and is efficient to optimize in the linear regression case . For intuition , consider a single-point distribution that always outputs labeled examples of the form ( x , y ) , where the example x is always the same but the labels y may differ . The Massart assumption indicates that the value of y is correct more than half of the time , so the estimate that maximizes the number of correct samples ( ` 0-minimizer ) recovers the underlying function . However , if one considers the ` 1-minimizer , i.e. , the value v that minimizes E [ |y v| ] , this corresponds to the median value of y which is also correct if more than half samples are correct . We generalize this intuition and propose a natural and tight condition under which empirical ` 1- minimization results in the true ` 0-minimizer ( see Lemma 2.2 ) . While this condition holds under Massart noise for arbitrary distributions in the population level , it can fail to hold with high probability when considering only a finite set of samples from the distribution . For example , consider the onedimensional case of w⇤ = 1 where most xi ’ s are near-zero and uncorrupted , while a few corrupted samples lie extremely far from zero . The empirical ` 1-minimizer here will be dominated by the few corrupted samples and would not be the ` 0-minimizer . In particular , the sample complexity of naive ` 1-minimization would crucially depend on the concentration properties of the distribution on x . Transforming the points via radial-isotropy The main technical idea behind obtaining sampleefficient algorithms that run in polynomial time is to transform the original point set into an equivalent one that satisfies the required properties with high probability , as it becomes sufficiently concentrated . In particular , performing a linear transformation mapping every point x to Ax , while keeping the corresponding label y , is without loss of generality , as we are interested in identifying the true ( generalized ) linear function that depends only on the inner product of every point with a parameter vector w. Finding such a vector w0 in the transformed space Ax results in the equivalent vector w = ATw0 in the original space . Moreover , an additional operation we can perform is to take a single sample ( x , y ) and multiply it by a positive scalar > 0 to replace it with the sample ( x , y ) . For both the linear and ReLU cases , any sample that is an inlier for the true function remains an inlier after this transformation . We can use these two operations to bring our point set in radial-isotropic position , i.e. , so that all the x ’ s in the dataset are unit-norm and the variance in any direction is nearly identical : Definition 1.4 ( Radial Isotropy ) . Let Sd 1 be the unit sphere in Rd . Given { x1 , . . . , xn } ⇢ Sd 1 , A : Rd ! Rd is a radial-isotropic transformation if Pn i=1 ( Axi ) ( Axi ) T kAxik22 = nd I . For 0 < < 1 , we say the points are in -approximate radial-isotropic position , if for all v 2 Sd 1 , it holds that ( d/n ) Pn i=1 ( xi · v ) 2 1 . In such a normalized position , we can argue that with high probability the weight of all inliers in every direction is more than the weight of the outliers , which guarantees that the empirical ` 1-minimizer will converge to the true function . Learning ReLUs Unfortunately , while ` 1-minimization for linear functions is convex and efficiently solvable via linear programming , ` 1-minimization for ReLUs is challenging due to its non-convexity ; that is , we can not easily reduce ReLU regression to a simple optimization method . We instead establish a structural condition under which we can compute an efficient separation oracle between the optimal parameter vector w⇤ and a query w. More specifically , we show that any suboptimal guess for the parameter vector w can be improved by moving along the opposite direction of the gradient of the ` 1-loss for the subset of points in which the condition in Lemma 3.1 is satisfied . Identifying such a direction of improvement yields a separating hyperplane , so we exploit this to efficiently identify w⇤ by running the ellipsoid method with our separation oracle . However , for this result to hold with a small number of samples , we need to again bring to radialisotropic position the points that fall in the linear ( positive ) part of the ReLU for the current guess vector w. In contrast to the linear case , though , where this transformation was applied once , in this case it needs to be applied again with every new guess . This results in a function that changes at every step , which is not suitable for direct optimization . Using these ideas , our algorithms can efficiently recover the underlying function exactly using few samples . Our algorithms make mild genericity assumptions about the position of the points , requiring that the points are not concentrated on a lower-dimensional subspace or , for the case of ReLUs , do not lie entirely in an origin-centered halfspace . As already mentioned , such assumptions are necessary for the purposes of identifiability .

Summary: The aim of this work is to propose a computationally efficient algorithm for linear and ReLU regression in the presence of a bounded adversarial Massart noise model. It has been tried to make the assumptions on the underlying data distribution as mild as possible, not to mention that (as claimed by the author(s)) some assumptions are necessary from an information-theoretic point of view.  The core idea is to consider the unknown corrupted samples as outliers which do not follow the simple linear or ReLU rules between their features and corresponding labels. This should be added to the fact that (based on the paper's assumption) the fraction of corrupted samples is less than 1/2, so the clean data points have the majority. This way, author(s) have proposed a linear transformation on the samples in order the make them "Radially Isotropic", so (with high probability) the ratio of outliers to clean samples remain as low as possible in *every direction* of the space. Then, some existing robust estimators such as sum of $\ell_0$ or $\ell_1$ models have been shown to remove the effect of outliers and acquire the true parameters. I haven't completely checked the proofs, however, the overall idea makes sense to me and is, in fact, quite interesting. In any case, I am not completely familiar with this line of research so I wait to see other reviews in order to assess the novelty of techniques that have been utilized in this work.  With respect to weaknesses, paper lacks proper discussion at some points which I have explained in the "Main review" section. Also, author(s) have assumed a completely clean and noise-free model except for the adversarial part which still includes more than half of the samples. This assumption is a little worrisome in practice, where some minimum levels of, for example, additive noise are *always* present. How robust or sensitive is the proposed method and its theoretical guarantees when the presumed ideal noise-free environment is minimally perturbed?  Overall a well-motivated and fairly well-written paper with some interesting theoretical achievements (as far as I am aware). My vote at this stage is weak-accept.